Category 7: Measuring Effectiveness

 

7P1Selection, Management and Distribution of Data.Various processes are used to select performance data including internal administrative identification of key indicators for the college, State of Michigan Performance Indicators, Federal Perkins Core Indicators, and the Federal IPEDS Data System. Management of information is coordinated primarily through the Institutional Research (IR) office and the Accounting/Finance office. Distribution of data is accomplished by way of employee forums, web pages, memos, e-mails and submission/participation in external reporting sites. The Director of Institutional Research maintains a taxonomy of all reports here:
http://www.mcc.edu/mccfact/pdf_mccfact/ReportTaxonomy.pdf

  • Video

7P2Data Support for Planning and Improvement. Data are selected based on a number of criteria for collection and distribution. One of the major criteria for selection of data is internal use by administrators, faculty and staff in the management of college processes and improvement efforts. Another key criterion is State and Federal reporting. A comprehensive list of 254 standard data reports may be found in the Report Taxonomy described above in 7P1. A selected list of the information tracked on standing data reports in the Taxonomy appears in the following table:

Taxonomy Field Name Field Contents/Examples
Report Name Career Education Consumer Report, Economic Development Job Training Audit (EDJT), HLC Organizational Profile, Institutional Characteristics (IPEDS), Program Review of Occupational Education (PROE), Graduation Rates, HLC Financial Ratios, Institutional Snapshot, Performance Measures for Community Colleges, Retirement Detail Report, Student Right to Know, Tech Prep Report, Year-end Program Enrollments (254 total reports)
Purpose/Definition Contains a short phrase defining the report and its intended use
Department/Position Responsible Institutional Research, Accounting/Finance, Human Resources, Foundation for MCC, Fine Arts Division, etc.
Relationship to Strategic Plan Lists overarching goals from current strategic planning document by number (examples: 6-2, 1-3, 3-3, 5, etc.)
Report Category Institutional Information, Finance, Employee Information, Enrollment, Awards, Program Status, ACS, Aid Award, Perkins Data, etc.
Primary Users Internal, External, HR Staff, Faculty, Administration, Fine Arts Division, Students, Upward Bound Planning, Career & Technical Education, etc.
How Utilized Program Development, Cancelled courses—No activity, Planning Staff Assignments, Student Demographics, Cohort Counts, Course Change Impact, Prerequisite Courses, Planning/Budget, etc.
Report Audience WIA Clients, State of Michigan, National Center for Education Statistics, Student Services, US Department of Education, etc.
Unit/Analysis Student Cohorts, Courses, Students/Financial, Seat Counts, Buildings, Employee, Degrees, SIEF Survey, Award, Faculty, High Schools, etc.
Figure 7-1 List of Data Selection Fields in MCC’s Report Taxonomy (selected)

The taxonomy also contains fields identifying report aliases, ID numbers, external web links, submission methods, as well as requestor, agency, and cycle indicators. Annual department-level actions plans are prepared, approved, managed and assessed by internal administrators. These action plans are aligned with the MCC strategic plan and other institutional priorities, such as AQIP Action Projects.

In addition to internal and external standing reports, MCC’s Assessment Coordinator regularly shares student learning data at faculty meetings, including the annual Assessment Update. Assessment data are also shared online here:
http://www.mcc.edu/acad_affairs/aa_gen_ed.shtml

7P3Collection, Storage, and Accessibility of Data. A large MCC Datatel Users Group (MCCDUG) is chaired by the Chief Technology Officer (CTO) and meets regularly to determine departmental need for data collection, storage, and accessibility. MCC maintains a data warehouse with archived “snapshots” of data at specific key dates during the fiscal and academic year. Business intelligence software (currently IBM’s Cognos) is used to query either warehouse data for historical/longitudinal reporting or live data in Datatel Colleague. A more detailed explanation of MCC’s data warehouse architecture appears in 5P6. The institutional Report Taxonomy, maintained by Institutional Research, documents 254 external and internal information requirements; custom reports are developed for particular requests by managers, project leaders, and other internal stakeholders.

The storage of source and final data is done in secured IR office storage and on password-protected institutional electronic storage. Access is available to the Planning, Research and Quality staff as well as the Information Technology Services (ITS) staff. Many data resources are maintained on the college web site for accessibility to both internal and external stakeholders.

7P4Analysis of Performance Data. A central design of the data warehouse model is to have one common data set that can be analyzed at the department level by individual leaders and managers who track and chart progress in overall performance. MCC has not yet developed dashboards or balanced scorecards based on this reporting ability, but the capacity to do so now exists and discussions are planned to proceed in this direction. Executive Cabinet members and Board of Trustees members review and analyze “big picture” data regarding the college’s finances, enrollment, infrastructure and stakeholder satisfaction. These analyses are distributed across the institution by way of Board Meeting minutes and web pages. Department level managers use these information sources as well as their own measurements such as seats filled, revenue and expenditure at the program/discipline level, and student satisfaction and assessment data at the program and course levels. Data is maintained and openly-available on college web pages, college network servers, and Datatel Colleague/Cognos systems and is made available to employees and the community.

7P5Comparative Data. The needs and priorities for comparative data are driven by environmental scanning—both formal and informal—to create a context for data that reflects our service area and the unique needs of the district and contiguous communities. State and Federal comparisons allow for identification of areas to strengthen, improve or discontinue. State comparative data books and Federal IPEDS peer group analysis, as well as state AQIP group members provide benchmark criteria against which institutional data are evaluated. Local area socio-economic factors such as changes in population and the labor market and employer needs set the primary context for data to be used. College representatives—including the Director of Intuitional Research, the Chief Technology Officer, the Vice President for Academic Affairs, the Executive Dean of Planning, Research, and Quality—serve on state-wide data task forces and work groups.

MCC is considering joining the National Community College Benchmarking Project (NCCBP). Work teams and individuals connected with measuring effectiveness at the college have a number of touch points with Johnson County Community College where the NCCBP is housed. Attendees to the 2009 HLC Annual Meeting attended a session on the NCCBP, and MCC’s participation is currently under consideration in the annual budgeting process.

Comparative data are also available through institution-wide assessments such as MAPP results, which are discussed in greater detail in 1R1, 1R2, and 1R6.

7P6Alignment of Data Analysis With Organizational Goals. All department administrators and many other department staff have direct access to standard and ad hoc reports in Datatel Colleague and Cognos. Lead administers set direction for departments to analyze and compare their local data to college-wide expectations. An example is the budget modification process; institutional targets are set for reductions and department data are aligned to accomplish the necessary cuts.

7P7Information Systems and Processes. MCC invests significant human and technological resources to ensure the timeliness, accuracy, reliability, and security of information systems. An entire section of MCC’s strategic plan is devoted to technology-related initiatives:

2-0 Technology Initiatives
2-1 Commit the funds, including staffing costs, to continuously analyze and support the technology needs both in the teaching curriculum and the college’s operations and student services.
2-2 Maintain state of the art technology that enhances student learning and supports faculty/staff productivity in order to maximize student success and organizational effectiveness.
Figure 7-2 2007-2012 Overarching Goals for Technology Initiatives

MCC maintains powerful systems for managing and analyzing information. The ITS department has the accountability to purchase, install, configure and maintain the primary college information systems; the Executive Cabinet routinely reviews systems needs and usage and makes recommendations to the Chief Information Officer (CTO) for changes and improvements. Overarching Goals and Enabling Objectives in the Strategic Plan set data standards and development objectives.

7R1Measures of Knowledge Management. Datatel Action Planning and Cognos Systems Utilization studies provide benchmarks. The Datatel Action Planning Process currently underway is outlined in detail below in Figure 8-3. ITS staff regularly conduct performance audits and usage reports for their analysis of system performance and effectiveness. Data standards are established and upheld by groups of trained staff.

7R2and7R3Results for Measuring Effectiveness. Members of the Executive Cabinet have varied methods for understanding and using system tools and data. External reports are completed in timely and accurate fashion as described in 7P1 and 7P2 above. Student and Graduate satisfaction measures (described in 3R1 and 3R2) are analyzed to identify institutional processes that need to be improved. In addition, ITS conducts user groups and service evaluation questionnaires. Other system processes such as the Office of Physical Plant service requests are analyzed for effective outcomes. The Director of Institutional Research (IR) and the Chief Technology Officer (CTO) participate in state and national groups that serve to develop and improve system processes such as data warehouse utilization.

As evidenced by other sections of this portfolio (see answers to 4R4, 5R3, and 6R5 for examples), benchmarking external performance results is an area of relative weakness for certain processes at MCC. As part of our participation in MiTQIP, MCC participated in a pilot benchmarking project on developmental education measures. The table below shows the preliminary results of this effort. It should be noted that community colleges in Michigan have disparate policies and practices in regard to developmental writing, reading, and math. In addition, a few of the institutions represented in the data set—including MCC—do not have mandatory placement policies. The developmental benchmarking project is one of many that MCC plans to undertake with other MiTQIP institutions in the future.

Developmental Writing Fall 2005 Winter 2006 Fall 2006 Winter 2007
Glen Oaks Community College N/A N/A 56% 71%
Jackson Community College 85% 79% NA NA
Kirtland Community College 77% 78% 77% 84%
Mid-Michigan Community College* 65% 67% 60% 55%
Mott Community College* 87% 72% 75% 72%
West Shore Community College 86% 69% 76% 68%
Developmental Reading
Glen Oaks Community College N/A N/A 83% 80%
Jackson Community College 81% N/A NA NA
Kirtland Community College 57% 82% 64% 91%
Mid-Michigan Community College* 85% 61% 64% 75%
Mott Community College* 61% 78% 77% 78%
West Shore Community College 83% 76% 78% 67%
Developmental Math
Glen Oaks Community College N/A N/A 76% 66%
Jackson Community College 69% 73% NA NA
Kirtland Community College 66% 61% 61% 84%
Mid-Michigan Community College 62% 64% N/A N/A
Mott Community College* 71% 64% 70% 59%
West Shore Community College 72% 60% 75% 75%
* NO mandatory placement.       7/24/07
Figure 7-3 Pass Rates for Developmental Courses at MiTQIP Colleges

7I1Recent Improvements in Measuring Effectiveness. After many years of operating two separate departments charged with maintaining organizational computing technology, MCC worked to merge the previous Management Information Systems (MIS) and Educational Technology (EdTech) areas in June of 2008. The two areas were combined to form Information Technology Services (ITS) under the direction of a Chief Technology Officer (CTO). While the new department is organized in the area of Academic Affairs, it services the entire institution. In the past, the split between MIS and EdTech functions was emblematic of “silos” at MCC; faculty and staff worked within separate e-mail systems; classroom support and administrative support were completely separate processes.

On August 26, 2008, ITS held a retreat to kick off the Fall 2008 semester. There were two objectives for the retreat: 1) foster communication and teamwork with the newly-formed department, 2) discuss the service goals of the department with respect to the college community.

The first half of the retreat was spent getting to know one another and discussing the structure of the department, particularly how the processing of service requests from both college employees and students. Before the merger, support for classrooms and students were handled by Educational Systems and support for staff and faculty office or administrative needs were handled by Management Information Systems (MIS). All calls for service—including computer, AV and network—are now directed to a centralized function entitled Computing Support Services (CSS). The group discussed expectations and request processing, in addition to physical residency ideas on how to provide seamless support for all users. The second half of the retreat was a structured group activity, where each group was to create a video that would explain to the college community how this new department would serve the college. These video presentations efficiently explain the cultural shift underway in re-organizing the information technology functions at MCC and can be viewed here:
http://www.mcc.edu/its/its_staff_dev_activities.shtml

7I2Improvement Efforts for Measuring Effectiveness. The culture and infrastructure of MCC is moving toward greater reliance on a systems orientation to performance improvement. A growing number of departments and areas routinely develop goals, and priority is given to efforts to improve systems, infrastructure, and activities leading to improved student success and service to the community. These institutional strategies undergo periodic/annual review and modification. MCC needs to move beyond the AQIP Action Project process to identify opportunities to utilize CQI principles for processes improvement. There is a great deal of growth in the area of Measuring Effectiveness in MCC’s processes. Goals are being identified in a greater number of campus projects. Still, many of these goals are implicit or poorly defined; means of measuring progress toward targets is often done after the fact as opposed to being part of a clearly-defined improvement process.