Relevant and Irreverent News

Student Management Systems – some more thoughts

In a recent post I wrote about changing student management systems. Since then I have talked with a lot of people and done a lot of thinking. Here are some more thoughts.

At the risk of repeating myself I should remind you that the data in your current system is YOURS and you must arrange to secure it before moving to another system. Here is why.

  • There is a distinct possibility that your new SMS supplier will be unwilling (or unable) to transfer a reasonable amount of historical data to their system. Or, at the very least will charge extra for data that goes back more than a couple of years.
  • Don’t forget to ask about any archived student data. Often that can reside in a separate database that could get overlooked in any data conversion.
  • Without that data you won’t be able to produce transcripts for students who studied with you in the past.
  • Without that data you may not be able to provide relevant data to TEC (audit) or NZQA (EER) when requested.

The Cloud is a wonderful thing. It can make your system accessible anywhere, anytime, and on any platform (Windows, IOS, Android, Linux). It can ensure that your data is reliably and frequently backed up.

The key word here is “reliably”. How would you feel if you got a message from your SMS supplier saying that you system would not be available because of an outage of their Cloud management, or because system maintenance is required urgently? That would be especially upsetting if that happened on a date that you were due to submit a Mix of Provision, a Fees-Free report or some other important return.

Ask about their uptime guarantees (SLA) and what guarantee that the SMS will be available when you need it. It is an impossibility to have a 100% uptime guarantee. If that is being promised, then what else is the SMS provider promising that they cannot deliver on?

So, when evaluating an SMS to be delivered via the Cloud check very carefully the credentials of the Cloud service provider. Ask about when there was last an outage (if any) and ask about the frequency of problems.

Can we trust Workspace2?

I’m afraid that the answer to this question has to be “No”.

You may recall the problem I wrote about recently where the Fees-Free validation report had duplicate records for some providers and had to be re-released. That gave us a clue that all was not well in Workspace2.

The use of Workspace2 in relation to Fees-Free seems to be particularly badly affected as you will see from the following list.

  • A couple of weeks ago there were “intermittent login issues with Workspace 2”. You simply couldn’t login meaning that the template for the May Fees-Free return was not available.
  • Then when we did get the template and wanted to submit it we were told “We are experiencing intermittent login issues with Workspace 2. If you are unable to login, please contact the TEC Customer Contact Group on 0800 601 301 or email sectorhelpdesk@tec.govt.nz”.
  • The turnaround time for specific Fees-Free status requests is poor, apparently because they are processed manually and therefore only on working days.
  • The following message was sent out on 12 April. “Failure to load accurate Fees-Free data in Workspace 2. Please note: If you downloaded these files on April 10 or 11 please delete these as they contained errors. All learners that had their eligibility status changed as part of the statutory declaration process had their eligibility reset to ‘Unknown’ in error. The corrected versions are now published above.
  • Then there are the other discrepancies relating to students’ Fees-Free Eligibility status, for example: Can TEC explain how NSN nnnnnnnnnnn appears in 2018 Fees Free Consumption.csv and FeesFreeEligibility2019.CSV downloaded from Worksapce2 on 16-05-2019, but in the FeesFreeEligibility2019.CSV file the Fees Free Eligibility Flag is “Y” and not “8”? Apparently TEC cannot explain it because this question put to them on 17 May has yet to be answered.
  • The April Fees Free Multiple Providers and Cap Limit Report took a month to be released and now that it has, the report has invalid data in two columns.

Unfortunately, it is not just Workspace2 that is problematic. Have you tried to use the Search function on TEC’s web site? If you have, you were probably frustrated.

Often it doesn’t provide a response for even a straightforward search word or phrase. Other times it will provide several responses in a list but then not let you click through. I have been told that it is better just to use a Google search site specific search. For example, typing “Brendan Kelly site:tec.govt.nz” into a Google search will produce far more hits than entering “Brendan Kelly” into the search box on the TEC website. That tells you it has found 9 results but shows only 8.

Heaven protect us if any replacement for the SDR makes use of Workspace2 as a means of transferring files.

Overall it seems to me that there are a number of issues relating to TEC’s technology which come from poor business analysis, resulting in badly designed systems, further impacted by a lack of testing; but I stand to be corrected.

Student Management Systems – Time for Change?

What do you do when you want a new and better Student Management System; or possibly your current Student Management System (SMS) supplier goes out of business, or ceases to support your system? Here are some thoughts.

At present there seems to be a bit of action amongst the SMS suppliers which, having had a few anxious phone calls and emails, has prompted me to write this post.

Your current system

  • First and foremost, you should secure your data. Your existing SMS supplier will have licensed you to use their software and that licence may be being terminated, but remember the data is yours. Now, here’s the interesting and rather worrying thing. In the rush into the Cloud you may have lost control of your data.
    • If you are running your SMS on your own network your database should be simple to secure – indeed you may have multiple back-ups of that data going back years. There is no real cost to you to securing your data.
    • If your database is in the Cloud but stored by an organisation other than your SMS provider you should be able to retrieve your database yourself or request the supplier to send it to you. There may be a small cost associated with this service.
    • If your database is stored by your SMS supplier you may have a problem, depending on how that supplier manages data. It could be a simple matter for the supplier to pass you a copy of the physical database as a single file or as a structured group of files. In some instances, however your data may be stored in a “multi-tenanted” database. This means your data is stored alongside the data of other organisations using the same SMS. Then there may be a more significant cost to obtain your own data.
  • Another factor to consider is that when you get your database you may need to obtain a software licence to access the data and if that were, say, an Oracle licence this will be expensive. If the required licence is for a Microsoft product you may already have a licence or can acquire one easily.
  • You should check on your right to continue to use your current SMS. Some systems are supplied on the basis that you pay a fee to use the software – possibly on a monthly basis. If you cease to pay that fee your access to the software will be terminated. This mean that, even if you have secured your data, you can’t actually use it to do things like produce a Single Data Return. Other systems are supplied on the basis that you are provided with the software and can continue to use it even after the supplier has gone out of business or ceases to support the software. Clearly if the software is not updated for changing compliance requirements it will quickly become redundant; however, you can still use it to produce historical reports and academic transcripts. This is particularly important if moving to a new SMS involves a poor or limited transfer of data from your existing system.

A replacement SMS

  • If you are in the situation of having to change to another system you may find a checklist useful. Please contact me if would like a checklist, or just to chat about the process of change. In the meantime, here some ideas that may help.
  • Some people who market and sell systems don’t necessarily know very much about either the technology of the system they are selling nor, for that matter, the nuts and bolts of student management. They possibly also have little insight into the compliance requirements of New Zealand government agencies.
  • Make a list of the key features of your current SMS, including compliance functions, and check that they are in any product you are evaluating.
  • When assessing systems, you should ask to talk to the people who will provide support. You should also ask to view a sample of the user documentation. After all a system is only as good as the support that is provided.
  • Check that the underlying software used by the new system is not out of date or possibly even redundant. For example, Adobe will not support Flash beyond 2020.
  • Evaluate if having custom functionality is something that you might require. Some SMS suppliers do not allow custom functionality, or even allow you to access your database preventing you being able to use or develop third-party applications accessing your data. If the SMS supplier provides an API interface to exchange data, you need to evaluate how extensive it is and if it would meet your requirements. In most cases the API interface is limited and at the whim of the SMS supplier to expand.
  • Ask for references, making sure that you get several from providers who recently changed to the SMS you are checking out. You could also ask the supplier to name an organisation which has used their system but decided to move on to another product.
  • Get very clear and detailed information on pricing over say, a five-year horizon. At least one supplier has a built-in annual fee increment based on the consumer price index. Several suppliers have a separate fee for particular options. You may need to have functionality that includes the exchange of data with the Literacy and Numeracy for Adults Assessment Tool or reporting to Public Trust for example, but this could cost extra. You may want good business intelligence functionality, but this could require you to pay a third-party software licence.  
  • Check that you can have access to a test system with a copy of your data at no additional licence cost.
  • Check out in detail how the data from your current SMS gets into the new SMS. Who does the work? You may end up copying a lot of data into a bunch of CSV files. That takes time and can result in transposition errors. How far back in time will your data be transferred? You might have got records going back to the introduction of the SDR in your current system which are still of value – not least because ex-students want to get transcripts.
  • If though, the transfer only goes back one or two years, will there be a cost in maintaining your existing SMS in order to produce those transcripts?
  • Investigate the likely longevity of the supplier you are evaluating. It is not unknown for a supplier to enter the market in New Zealand and then discover that the market is not sufficiently large to earn enough to maintain their SMS’s New Zealand compliance. Annual fees are increased to close the gap or the supplier simply leaves New Zealand.
  • Ask for a copy of the current SMS development road map. This might provide an insight into missing or “thin” functionality.


Fees-Free – An Unfortunate Series of Events

Sadly, the implementation of Fees-Free is not an engaging set of novels for the younger reader however, like the books it has “a dark, mysterious feeling”. One is never sure what will happen next.

And it was whilst pondering the latest debacle (being a fail by TEC with the preparation and distribution of the “Fees Free All Enrolments and Costs Data Validation – March 2019” file), that I decided that I would try to document the set of processes used by TEC to manage Fees-Free and to enumerate the causes of its failure.

A consequence of that failure is that you cannot know how much your Fees-Free payment from TEC will be, you are likely to be non-compliant with TEC’s funding conditions, and your compliance costs will have increased significantly.

If you would like to read this sorry story, please let me know. You may even find it useful when explaining to your boss why you can’t provide an accurate report of expected income, or when dealing with auditors.

Oh, and the most recent fail with the Validation Report? Well if the problem was restricted to the two new (and undocumented) fields in this report it will cause you confusion but no hurt. Still, if TEC gets something this simple wrong you have to wonder what other unannounced problems there are with TEC’s reporting processes.

SDR Validation Error 143

If you are being a good provider and following TEC’s advice to carry out a trial SDR well in advance of the April SDR you are likely to fall foul of Validation Error 143. This error is explained as “IWI is not valid”. The problem you will find is that even if you have reported a valid Iwi code – i.e. one listed in the 2019 SDR Manual Appendix – you still get error 143.

The TEC Sector Helpdesk explains that this is a known bug and will be rectified by 5 April. When I suggested to the Helpdesk that it could lighten its load by publicising the problem, and thus reducing the number of calls it receives. This idea was not warmly welcomed, and it was suggested that the Ministry of Education was responsible for the SDR: buck passed.

Again, one is prompted to suggest that the title “Helpdesk” is something of a misnomer. TEC really doesn’t care if providers waste time proving to themselves that they are not going crazy, and that the problem lies instead with the validation software. Meaning that the providers then have to call the Helpdesk and, in all likelihood their calls won’t go through because the Helpdesk is overloaded.

In short TEC’s basic principle of “less trust, higher compliance cost” is still in effect.

How’s this for a plan

You may recall that in my last post I pointed out that the latest Youth Guarantee funding conditions are not workable.

The 2019 funding conditions 4.1 and 5.4 contradict each other.

  • 4.1 defines 1 EFTS as 100 credits.
  • 5.4 defines 1 EFTS as 120 credits.

This bizarre situation is as a result of a well-meaning but flawed policy change designed to deliver 20% more funding to YG providers. However, in practice it will significantly reduce funding for certain YG providers – especially the more successful. I know of one case where it could result in a cut of nearly 60%.

TEC has been promising to resolve the anomaly since December 2018 but, despite acknowledging that a solution is urgent, there has been no action.

This is where my good idea comes in. YG has been around since 2012. TEC took until 2018 to recognize that funding was insufficient. So, pending finding a way of delivering 20% more funding without disadvantaging some providers, TEC should make a one-time payment equal to 20% of the funding delivered to providers from 2012 to 2018.

After all TEC is very quick to claim back overfunding. It would be only fair for TEC to make up underfunding.

Can TEC Survive Much Longer?

I am not noted for my warm feelings towards TEC, but the opinion piece by Tina Nixon in the Wairarapa Times Age was a doozy and in a league of its own. You can read it yourself, but here are just a couple of samples.

  • TEC is without a doubt one of the most bureaucratic organisations I have ever interacted with, and I have worked with a few.”
  • If the TEC and its current administration survive the next year, then this government will have failed the sector.”

I can’t compete with that, but I can supply a few thoughts on TEC’s current performance – or lack thereof. In all cases the problem seems to relate to a lack of consistency on the part of policy makers. One condition/policy/requirement contradicts another.

Fees Free

Admittedly TEC had very short notice to implement the Fees Free policy but over a year has gone by and they are still struggling. The wash-up return for 2018 was initially scheduled for 11 January 2019 but after receiving a lot of complaints the return date was extended to 16 January. This is where it gets tricky because the Fees Free return will not necessarily match data to be reported through the SDR which is not due until 31 January 2019. For example, a student who started in 2018 with a programme running into 2019 notifies the provider on 17 January that they will not be returning – i.e. they are withdrawing. The provider can record that for the SDR but it will have missed the Fees Free return on 16 January.

Youth Guarantee

The 2019 funding conditions 4.1 and 5.4 contradict each other.

  • 4.1 defines 1 EFTS as 100 credits.
  • 5.4 defines 1 EFTS as 120 credits.

 Go figure!

Until this issue is resolved YG providers would be ill-advised to start enrolling students because they may inadvertently contravene other YG funding conditions.

Funding Source 03 – Domestic Full Fee Paying Students

The Single Data Return Manual states that all students must be reported through the SDR, including non-funded students. However, from the 2018 version 2 edition of the SDR manual the use of Funding Source 03 – Domestic Full Fee Paying Students was changed to exclude students validly repeating courses already completed in a TEC funded programme. This is a practice that was applied to a student who withdrew from a programme several years ago but who returns to complete it and – at their own wish and the wish of the training provider – wants to repeat the few courses already successfully completed, paying a fee to do so. There is now no funding source code that can be applied to these enrolments and so how can they be reported?

RIP Performance Linked Funding

On 7 November TEC announced that Performance Linked Funding (PLF) is to be discontinued. Instead TEC will continue “to use a range of levers to encourage better educational performance and address poor performance.”

So, rest in peace PLF. It has been a long and complex exercise which appears to have achieved very little except increase compliance costs for providers.

One recalls the Ministry of Education’s Statement of Service Objectives (SSO)/Statement of Service Performance (SSP) regime that was in place before TEC’s dead hand reached out for new levers. That regime involved setting two types of objectives: those for equal educational opportunity and those for educational performance. This seems to have covered a wide range of factors in a simple manner and without all the expenditure of ineffective initiatives such as MyQ.

But SSO/SSP morphed into KPIs (Key Performance Indictors) which went through various revisions before finally being replaced by EPIs (Education Performance Indicators).

So, what went wrong?

There seem to have been a number of factors:

  • Too many CEOs at TEC over the years, and a high turnover of those charged with policy making.
  • An inability to clearly define performance indicators: remember it took eight versions published from 2010 to 2014 to get to a definitive set of definitions and methodology for EPIs. No sooner had version 8 been released than the cohort-based indicators were being developed and promulgated – again with several revisions.
  • There has been an ongoing tinkering with the relationship between performance and funding, most recently exhibited by an unannounced trial of points-based EPI modelling.
  • There has been an increasing obfuscation of the calculation and publication of EPIs on the part of TEC. What were useful tools in Workspace2 became a nightmare in Nga Kete.
  • Nga Kete’s interface reflects an overfondness for technology that seems more aimed at satisfying TEC’s internal requirements (and IT predilections) than meeting the needs of providers.

This said, it must also be pointed out that The Ministry of Education’s SSO/SSP regime relied on attestations which were not rigorously monitored. Over the years some providers found how to game the system. As they were found out increased compliance measures have had to be introduced. PEI’s and Intueri’s actions were the most notable instances.

As Roy Sharp (TEC’s CEO back around 2010) observed, there is a balance between trust and compliance costs. His inclination being to place faith in trust, a position that did not last long after his brief tenure  ended.

Experienced staff – not clanking robots

There is also a balance between a wholly mechanistic method of allocating funding, and a method which involves mature discretion on the part of experienced staff at the funding agency. Such staff do exist within TEC so hopefully it will be their hands on the levers in future and not some clanking robots.

[RH1]

Changing Student Management System?

It’s the time of year when providers that are thinking about changing student management system start to do something about it. If you are in this situation you may find the notes below useful.

As you will be aware I have been involved with student management systems for quite a while, and I was responsible for the development and support of one particular system from 1998 to early 2017. At the start of 2016 my business partner and I sold our interest in that product and I was employed by the new owner until February 2017. At that point I left to return to work as an independent consultant and, I must say, life without management meetings has been a glorious relief.

I have since had the opportunity of working with clients with other student management systems, or who are moving to other systems. This leads me to make some observations which may be relevant to you if you are thinking of changing.

  •  The people who market and sell systems don’t necessarily know very much about either the technology of the system they are selling nor, for that matter, the nuts and bolts of student management. They probably also have little insight into the compliance requirements of government agencies.
  • You should ask to talk to the people who will provide support. After all a system is only as good as the support that is provided.
  • Check that the software used by the system is not out of date or possibly even redundant; for example, “Flash”.
  • Where your organisation has multiple organisational units (campuses, departments, etc.) and/or EDUMIS numbers you will want what is called “multitenancy” software. A tenant is a group of users with specific privileges. For example, users at campus “A” can’t see student records at Campus “B”, however a head office user can see all students and produce reports across the whole organisation as well as for the individual campuses. Not all systems which claim multitenancy do in fact work in this way.
  • Nobody likes reading user manuals or online help (and system developers don’t necessarily like writing this material) but you should make a point of asking to see the material and check that it is up-to-date.
  • Make sure that you own your data and can access it independently of the student management system itself and the supplier of the system. This is particularly important if your data is being stored in the cloud.

EPIC: non-Māori, non-Pasifika

TEC’s EPIC template for 2019 is quite different from previous years. It omits an age focused goal (under 25 and 25+) but puts a lot more emphasis on ethnicity by introducing the concept of non-Māori, non-Pasifika.

Government and TEC’s policy is now focused on improving participation and achievement by Māori and Pasifika and therefore a means of accurately determining ethic rates is needed. With Statistics New Zealand’s move from “prioritised” to “total response” counting it is no longer possible to reliably express ethnic rates that sum to 100%.

Consider the tables below. There are 10 students.


The old “prioritised” method always resulted in being able to calculate a percentage for each ethnic group, and those percentages added to 100%.

The new “total response” method will always result in a percentage total of over 100% if any student identified with more than one ethnic group.

However when the count is binary. i.e. a student is either Maori or Pasifika if any one of their ethnicities is Maori or Pasifika, or they are non-Māori, non-Pasifika is none of their declared ethnicities is Maori or Pasifika, we once again have a total of 100%. Hence the table below for the same 10 students.