In a recent post I wrote about changing student management systems. Since then I have talked with a lot of people and done a lot of thinking. Here are some more thoughts.
At the risk of repeating myself I should remind you that the data in your current system is YOURS and you must arrange to secure it before moving to another system. Here is why.
There is a distinct possibility that your new SMS supplier will be unwilling (or unable) to transfer a reasonable amount of historical data to their system. Or, at the very least will charge extra for data that goes back more than a couple of years.
Don’t forget to ask about any archived student data. Often that can reside in a separate database that could get overlooked in any data conversion.
Without that data you won’t be able to produce transcripts for students who studied with you in the past.
Without that data you may not be able to provide relevant data to TEC (audit) or NZQA (EER) when requested.
The Cloud is a wonderful thing. It can make your system accessible anywhere, anytime, and on any platform (Windows, IOS, Android, Linux). It can ensure that your data is reliably and frequently backed up.
The key word here is “reliably”. How would you feel if you got a message from your SMS supplier saying that you system would not be available because of an outage of their Cloud management, or because system maintenance is required urgently? That would be especially upsetting if that happened on a date that you were due to submit a Mix of Provision, a Fees-Free report or some other important return.
Ask about their uptime guarantees (SLA) and what guarantee that the SMS will be available when you need it. It is an impossibility to have a 100% uptime guarantee. If that is being promised, then what else is the SMS provider promising that they cannot deliver on?
So, when evaluating an SMS to be delivered via the Cloud check very carefully the credentials of the Cloud service provider. Ask about when there was last an outage (if any) and ask about the frequency of problems.
I’m afraid that the answer to this question has to be “No”.
You may recall the problem I wrote about recently where the
Fees-Free validation report had duplicate records for some providers and had to
be re-released. That gave us a clue that all was not well in Workspace2.
The use of Workspace2 in relation to Fees-Free seems to be particularly
badly affected as you will see from the following list.
A couple of weeks ago there were “intermittent login issues with Workspace 2”.
You simply couldn’t login meaning that the template for the May Fees-Free
return was not available.
Then when we did get the template and
wanted to submit it we were told “We are
experiencing intermittent login issues with Workspace 2. If you are unable to
login, please contact the TEC Customer Contact Group on 0800 601 301 or email email@example.com”.
The turnaround time for specific Fees-Free
status requests is poor, apparently because they are processed manually
and therefore only on working days.
The following message was sent out on 12 April. “Failure to load accurate Fees-Free data in
Workspace 2. Please note: If you downloaded these files on April 10 or 11
please delete these as they contained errors. All learners that had their
eligibility status changed as part of the statutory declaration process had
their eligibility reset to ‘Unknown’ in error. The corrected versions are now
Then there are the other discrepancies relating
to students’ Fees-Free Eligibility status, for example: Can TEC explain how NSN nnnnnnnnnnn appears in 2018 Fees Free
Consumption.csv and FeesFreeEligibility2019.CSV downloaded from Worksapce2 on
16-05-2019, but in the FeesFreeEligibility2019.CSV file the Fees Free
Eligibility Flag is “Y” and not “8”? Apparently TEC cannot explain it because
this question put to them on 17 May has yet to be answered.
The April Fees
Free Multiple Providers and Cap Limit Report took a month to be released and
now that it has, the report has invalid data in two columns.
Unfortunately, it is not just Workspace2 that is problematic. Have you tried to use the Search function on TEC’s web site? If you have, you were probably frustrated.
Often it doesn’t provide a response for even a straightforward
search word or phrase. Other times it will provide several responses in a list
but then not let you click through. I have been told that it is better just to
use a Google search site specific search. For example, typing “Brendan Kelly
site:tec.govt.nz” into a Google search will produce far more hits than entering
“Brendan Kelly” into the search box on the TEC website. That tells you it has found
9 results but shows only 8.
Heaven protect us if any replacement for the SDR makes use
of Workspace2 as a means of transferring files.
Overall it seems to me that there are a number of issues
relating to TEC’s technology which come from poor business analysis, resulting
in badly designed systems, further impacted by a lack of testing; but I stand
to be corrected.
What do you do when you want a new and better Student
Management System; or possibly your current Student Management System (SMS)
supplier goes out of business, or ceases to support your system? Here are some
At present there seems to be a bit of action amongst the SMS suppliers which, having had a few anxious phone calls and emails, has prompted me to write this post.
Your current system
First and foremost, you should secure your data.
Your existing SMS supplier will have licensed you to use their software and
that licence may be being terminated, but remember the data is yours. Now,
here’s the interesting and rather worrying thing. In the rush into the Cloud
you may have lost control of your data.
If you are running your SMS on your own network
your database should be simple to secure – indeed you may have multiple
back-ups of that data going back years. There is no real cost to you to
securing your data.
If your database is in the Cloud but stored by an
organisation other than your SMS provider you should be able to retrieve
your database yourself or request the supplier to send it to you. There may be
a small cost associated with this service.
If your database is stored by your SMS supplier
you may have a problem, depending on how that supplier manages data. It could
be a simple matter for the supplier to pass you a copy of the physical database
as a single file or as a structured group of files. In some instances, however
your data may be stored in a “multi-tenanted” database. This means your data is
stored alongside the data of other organisations using the same SMS. Then there
may be a more significant cost to obtain your own data.
Another factor to consider is that when you get
your database you may need to obtain a software licence to access the data and
if that were, say, an Oracle licence this will be expensive. If the required
licence is for a Microsoft product you may already have a licence or can
acquire one easily.
You should check on your right to continue to
use your current SMS. Some systems are supplied on the basis that you pay a fee
to use the software – possibly on a monthly basis. If you cease to pay that fee
your access to the software will be terminated. This mean that, even if you have
secured your data, you can’t actually use it to do things like produce a Single
Data Return. Other systems are supplied on the basis that you are provided with
the software and can continue to use it even after the supplier has gone out of
business or ceases to support the software. Clearly if the software is not
updated for changing compliance requirements it will quickly become redundant;
however, you can still use it to produce historical reports and academic
transcripts. This is particularly important if moving to a new SMS involves a
poor or limited transfer of data from your existing system.
A replacement SMS
If you are in the situation of having to change to another system you may find a checklist useful. Please contact me if would like a checklist, or just to chat about the process of change. In the meantime, here some ideas that may help.
Some people who market and sell systems don’t necessarily know very much about either the technology of the system they are selling nor, for that matter, the nuts and bolts of student management. They possibly also have little insight into the compliance requirements of New Zealand government agencies.
Make a list of the key features of your current SMS, including compliance functions, and check that they are in any product you are evaluating.
When assessing systems, you should ask to talk to the people who will provide support. You should also ask to view a sample of the user documentation. After all a system is only as good as the support that is provided.
Check that the underlying software used by the new system is not out of date or possibly even redundant. For example, Adobe will not support Flash beyond 2020.
Evaluate if having custom functionality is something that you might require. Some SMS suppliers do not allow custom functionality, or even allow you to access your database preventing you being able to use or develop third-party applications accessing your data. If the SMS supplier provides an API interface to exchange data, you need to evaluate how extensive it is and if it would meet your requirements. In most cases the API interface is limited and at the whim of the SMS supplier to expand.
Ask for references, making sure that you get several from providers who recently changed to the SMS you are checking out. You could also ask the supplier to name an organisation which has used their system but decided to move on to another product.
Get very clear and detailed information on pricing over say, a five-year horizon. At least one supplier has a built-in annual fee increment based on the consumer price index. Several suppliers have a separate fee for particular options. You may need to have functionality that includes the exchange of data with the Literacy and Numeracy for Adults Assessment Tool or reporting to Public Trust for example, but this could cost extra. You may want good business intelligence functionality, but this could require you to pay a third-party software licence.
Check that you can have access to a test system with a copy of your data at no additional licence cost.
Check out in detail how the data from your current SMS gets into the new SMS. Who does the work? You may end up copying a lot of data into a bunch of CSV files. That takes time and can result in transposition errors. How far back in time will your data be transferred? You might have got records going back to the introduction of the SDR in your current system which are still of value – not least because ex-students want to get transcripts.
If though, the transfer only goes back one or two years, will there be a cost in maintaining your existing SMS in order to produce those transcripts?
Investigate the likely longevity of the supplier you are evaluating. It is not unknown for a supplier to enter the market in New Zealand and then discover that the market is not sufficiently large to earn enough to maintain their SMS’s New Zealand compliance. Annual fees are increased to close the gap or the supplier simply leaves New Zealand.
Ask for a copy of the current SMS development road map. This might provide an insight into missing or “thin” functionality.
Sadly, the implementation of Fees-Free is not an engaging set
of novels for the younger reader however, like the books it has “a dark, mysterious feeling”. One
is never sure what will happen next.
And it was whilst pondering the latest debacle (being a fail
by TEC with the preparation and distribution of the “Fees Free All Enrolments and Costs Data Validation – March 2019” file),
that I decided that I would try to document the set of processes used by TEC to
manage Fees-Free and to enumerate the causes of its failure.
A consequence of that failure is that you cannot know how
much your Fees-Free payment from TEC will be, you are likely to be non-compliant
with TEC’s funding conditions, and your compliance costs will have increased
If you would like to read this sorry story, please let me know. You may even find it useful when explaining to your boss why you can’t provide an accurate report of expected income, or when dealing with auditors.
Oh, and the most recent fail with the Validation Report? Well if the problem was restricted to the two new (and undocumented) fields in this report it will cause you confusion but no hurt. Still, if TEC gets something this simple wrong you have to wonder what other unannounced problems there are with TEC’s reporting processes.
If you are being a good provider and following TEC’s advice
to carry out a trial SDR well in advance of the April SDR you are likely to
fall foul of Validation Error 143. This error is explained as “IWI
is not valid”. The problem you will find is that even if you have
reported a valid Iwi code – i.e. one listed in the 2019 SDR Manual Appendix –
you still get error 143.
The TEC Sector Helpdesk explains that this is a known bug and will be rectified by 5 April. When I suggested to the Helpdesk that it could lighten its load by publicising the problem, and thus reducing the number of calls it receives. This idea was not warmly welcomed, and it was suggested that the Ministry of Education was responsible for the SDR: buck passed.
Again, one is prompted to suggest that the title “Helpdesk” is something of a misnomer. TEC really doesn’t care if providers waste time proving to themselves that they are not going crazy, and that the problem lies instead with the validation software. Meaning that the providers then have to call the Helpdesk and, in all likelihood their calls won’t go through because the Helpdesk is overloaded.
In short TEC’s basic principle of “less trust, higher compliance cost” is still in effect.
You may recall that in my last post I pointed out that the latest Youth Guarantee funding conditions are not workable.
funding conditions 4.1 and 5.4 contradict each other.
defines 1 EFTS as 100 credits.
defines 1 EFTS as 120 credits.
situation is as a result of a well-meaning but flawed policy change designed to
deliver 20% more funding to YG providers. However, in practice it will significantly
reduce funding for certain YG providers – especially the more successful. I know
of one case where it could result in a cut of nearly 60%.
been promising to resolve the anomaly since December 2018 but, despite acknowledging
that a solution is urgent, there has been no action.
where my good idea comes in. YG has been around since 2012. TEC took until 2018
to recognize that funding was insufficient. So, pending finding a way of
delivering 20% more funding without disadvantaging some providers, TEC should
make a one-time payment equal to 20% of the funding delivered to providers from
2012 to 2018.
TEC is very quick to claim back overfunding. It would be only fair for TEC to
make up underfunding.
I am not noted for my warm feelings towards TEC, but the opinion piece by Tina Nixon in the Wairarapa Times Age was a doozy and in a league of its own. You can read it yourself, but here are just a couple of samples.
“TEC is without a doubt one of the most
bureaucratic organisations I have ever interacted with, and I have worked with
“If the TEC and its current administration
survive the next year, then this government will have failed the sector.”
compete with that, but I can supply a few thoughts on TEC’s current performance
– or lack thereof. In all cases the problem seems to relate to a lack of consistency
on the part of policy makers. One condition/policy/requirement contradicts
TEC had very short notice to implement the Fees Free policy but over a year has
gone by and they are still struggling. The wash-up return for 2018 was initially
scheduled for 11 January 2019 but after receiving a lot of complaints the
return date was extended to 16 January. This is where it gets tricky because
the Fees Free return will not necessarily match data to be reported through the
SDR which is not due until 31 January 2019. For example, a student who started
in 2018 with a programme running into 2019 notifies the provider on 17 January that
they will not be returning – i.e. they are withdrawing. The provider can record
that for the SDR but it will have missed the Fees Free return on 16 January.
funding conditions 4.1 and 5.4 contradict each other.
defines 1 EFTS as 100 credits.
defines 1 EFTS as 120 credits.
issue is resolved YG providers would be ill-advised to start enrolling students
because they may inadvertently contravene other YG funding conditions.
Funding Source 03 – Domestic Full Fee Paying Students
The Single Data Return Manual states that all students must be reported through the SDR, including non-funded students. However, from the 2018 version 2 edition of the SDR manual the use of Funding Source 03 – Domestic Full Fee Paying Students was changed to exclude students validly repeating courses already completed in a TEC funded programme. This is a practice that was applied to a student who withdrew from a programme several years ago but who returns to complete it and – at their own wish and the wish of the training provider – wants to repeat the few courses already successfully completed, paying a fee to do so. There is now no funding source code that can be applied to these enrolments and so how can they be reported?
On 7 November TEC announced that Performance Linked Funding (PLF) is to be discontinued. Instead TEC will continue “to use a range of levers to encourage better educational performance and address poor performance.”
So, rest in peace PLF. It has been a long and complex exercise which appears to have achieved very little except increase compliance costs for providers.
One recalls the Ministry of Education’s Statement of Service Objectives (SSO)/Statement of Service Performance (SSP) regime that was in place before TEC’s dead hand reached out for new levers. That regime involved setting two types of objectives: those for equal educational opportunity and those for educational performance. This seems to have covered a wide range of factors in a simple manner and without all the expenditure of ineffective initiatives such as MyQ.
But SSO/SSP morphed into KPIs (Key Performance Indictors) which went through various revisions before finally being replaced by EPIs (Education Performance Indicators).
So, what went wrong?
There seem to have been a number of factors:
Too many CEOs at TEC over the years, and a high turnover of those charged with policy making.
An inability to clearly define performance indicators: remember it took eight versions published from 2010 to 2014 to get to a definitive set of definitions and methodology for EPIs. No sooner had version 8 been released than the cohort-based indicators were being developed and promulgated – again with several revisions.
There has been an ongoing tinkering with the relationship between performance and funding, most recently exhibited by an unannounced trial of points-based EPI modelling.
There has been an increasing obfuscation of the calculation and publication of EPIs on the part of TEC. What were useful tools in Workspace2 became a nightmare in Nga Kete.
Nga Kete’s interface reflects an overfondness for technology that seems more aimed at satisfying TEC’s internal requirements (and IT predilections) than meeting the needs of providers.
This said, it must also be pointed out that The Ministry of Education’s SSO/SSP regime relied on attestations which were not rigorously monitored. Over the years some providers found how to game the system. As they were found out increased compliance measures have had to be introduced. PEI’s and Intueri’s actions were the most notable instances.
As Roy Sharp (TEC’s CEO back around 2010) observed, there is a balance between trust and compliance costs. His inclination being to place faith in trust, a position that did not last long after his brief tenure ended.
There is also a balance between a wholly mechanistic method of allocating funding, and a method which involves mature discretion on the part of experienced staff at the funding agency. Such staff do exist within TEC so hopefully it will be their hands on the levers in future and not some clanking robots.
It’s the time of year when providers that are thinking about changing student management system start to do something about it. If you are in this situation you may find the notes below useful.
As you will be aware I have been involved with student management systems for quite a while, and I was responsible for the development and support of one particular system from 1998 to early 2017. At the start of 2016 my business partner and I sold our interest in that product and I was employed by the new owner until February 2017. At that point I left to return to work as an independent consultant and, I must say, life without management meetings has been a glorious relief.
I have since had the opportunity of working with clients with other student management systems, or who are moving to other systems. This leads me to make some observations which may be relevant to you if you are thinking of changing.
The people who market and sell systems don’t necessarily know very much about either the technology of the system they are selling nor, for that matter, the nuts and bolts of student management. They probably also have little insight into the compliance requirements of government agencies.
You should ask to talk to the people who will provide support. After all a system is only as good as the support that is provided.
Check that the software used by the system is not out of date or possibly even redundant; for example, “Flash”.
Where your organisation has multiple organisational units (campuses, departments, etc.) and/or EDUMIS numbers you will want what is called “multitenancy” software. A tenant is a group of users with specific privileges. For example, users at campus “A” can’t see student records at Campus “B”, however a head office user can see all students and produce reports across the whole organisation as well as for the individual campuses. Not all systems which claim multitenancy do in fact work in this way.
Nobody likes reading user manuals or online help (and system developers don’t necessarily like writing this material) but you should make a point of asking to see the material and check that it is up-to-date.
Make sure that you own your data and can access it independently of the student management system itself and the supplier of the system. This is particularly important if your data is being stored in the cloud.
TEC’s EPIC template for 2019 is quite different from previous years. It omits an age focused goal (under 25 and 25+) but puts a lot more emphasis on ethnicity by introducing the concept of non-Māori, non-Pasifika.
Government and TEC’s policy is now focused on improving participation and achievement by Māori and Pasifika and therefore a means of accurately determining ethic rates is needed. With Statistics New Zealand’s move from “prioritised” to “total response” counting it is no longer possible to reliably express ethnic rates that sum to 100%.
Consider the tables below. There are 10 students.
The old “prioritised” method always resulted in being able to calculate a percentage for each ethnic group, and those percentages added to 100%.
The new “total response” method will always result in a percentage total of over 100% if any student identified with more than one ethnic group.
However when the count is binary. i.e. a student is either Maori or Pasifika if any one of their ethnicities is Maori or Pasifika, or they are non-Māori, non-Pasifika is none of their declared ethnicities is Maori or Pasifika, we once again have a total of 100%. Hence the table below for the same 10 students.