catalyst / moodle-local_cohortauto

Automatically add users into cohorts. (Previously moodle-auth_mcae.)

Auto-cohort local plugin for Moodle 3.5+

This local plugin automatically adds users into cohorts, with names that derived from the users' profile fields.

This plugin requires Moodle 3.5 or greater (using PHP 7 or greater).

Auto-created cohorts are made in the top-level system context (CONTEXT_SYSTEM).

Migrating cohorts from the auth_mcae plugin

This plugin is derived from, and intended to deprecate, the earlier auth_mcae plugin by Andrew Kama. See the auth_mcae plugin repository for more information.

The two plugins could be installed side-by-side, but the local_cohortauto plugin can conflict with user update event handling from the earlier plugin, causing multiple cohorts to be created, each managed by its own plugin.

You can convert older cohorts to be managed by this plugin at: /local/cohortauto/convert.php

The MCAE plugin can be uninstalled before conversion takes place, but be sure to make a copy of any settings from the older plugin that you wish to keep.

Installation

• Download the archive and install the files into: your_moodle/local/cohortauto
• Ensure files have been copied in with the same permissions as the rest of your Moodle.
• Visit the Site administration - Notifications page and follow the instructions

Configuration and operation

Visit the *Site administration - Users - Accounts - Cohort role synchronization * page

Moodle uses the Mustache templating language for rendering data into HTML for presentation. This plugin reuses that functionality, by turning values from a user's profile into cohort names. This allows a site administrator to make rules for the automatic creation of cohorts based on user properties.

The process of creating automatic cohorts or adding users to them happens when each user profile is created or updated, or when command line scripts are run. (cli/sync_user.php for individuals, and cli_sync_users.php to sync across all users).

Main template (mainrule_fld)

One value per line.

In the template you may use any characters (except '{' and '}') and profile field values. To insert a profile field value, use a {{ field_name }} tag.

By default, Moodle provides the following profile fields:

{{ id }}, {{ auth }}, {{ confirmed }}, {{ policyagreed }}, {{ deleted }}, {{ suspended }}, {{ mnethostid }}, {{ username }}, {{ idnumber }}, {{ firstname }}, {{ lastname }}, {{ email.full }}, {{ email.username }}, {{ email.domain }}, {{ email.rootdomain }}, {{ emailstop }}, {{ icq }}, {{ skype }}, {{ yahoo }}, {{ aim }}, {{ msn }}, {{ phone1 }}, {{ phone2 }}, {{ institution }}, {{ department }}, {{ address }}, {{ city }}, {{ country }}, {{ lang }}, {{ calendartype }}, {{ theme }}, {{ timezone }}, {{ firstaccess }}, {{ lastaccess }}, {{ lastlogin }}, {{ currentlogin }}, {{ lastip }}, {{ secret }}, {{ picture }}, {{ url }}, {{ descriptionformat }}, {{ mailformat }}, {{ maildigest }}, {{ maildisplay }}, {{ autosubscribe }}, {{ trackforums }}, {{ timecreated }}, {{ timemodified }}, {{ trustbitmask }}, {{ imagealt }}, {{ lastnamephonetic }}, {{ firstnamephonetic }}, {{ middlename }}, {{ alternatename }}, {{ lastcourseaccess }}, {{ currentcourseaccess }}, {{ groupmember }}

The email field has 4 variants:

• {{ email.full }} - full address ([email protected])
• {{ email.username }} - only username (user)
• {{ email.domain }} - only domain (my.example.com)
• {{ email.rootdomain }} - root domain (example.com)

Additional tags become available if you have custom profile fields added. For example, if you were to create the following custom profile fields:

• checkboxtest - type Checkbox
• datetimetest - type Date/Time
• droptest - type Dropdown menu
• textareatest - type Text area
• textinputtext - type Text input

You would be able to use these tags:

{{ profile.checkboxtest }}, {{ profile.datetimetest }}, {{ profile.droptest }}, {{ profile.textinputtext }}, {{ profile_field_checkboxtest }}, {{ profile_field_datetimetest }}, {{ profile_field_droptest }}, {{ profile_field_textareatest.text }}, {{ profile_field_textareatest.format }}, {{ profile_field_textinputtext }}

Note: Profile field templates are case sensitive. {{ username }} and {{ UserName }} are two different fields!

Split arguments

Mustache also allows you to split a single field into multiple values.

Synopsis: %split(fieldname|delimiter)

This will return multiple cohort names, formed by splitting the field at boundaries specified by the delimiter.

• fieldname : Profile field name. The same as a tag, but without '{{' and '}}'
• delimiter : The boundary string. 1-5 characters.

Example:

User John fills the custom profile field "Known languages" with the value "English, Spanish, Chinese"

The main template contains a line: Language - %split(knownlanguage|, )

John will be enrolled in 3 cohorts:

• Language - English
• Language - Spanish
• Language - Chinese

Empty field text (secondrule_fld)

If profile field is empty, it will be replaced with this value.

Replace array (replace_arr)

If the values in your profile are expected to be different from what you want your cohort names to be, you can add replacement strings here.

One replacement per line, in the format: old value|new value

e.g.: Yoyodyne Propulsion Systems Incorporated|Yoyodyne

Note: Names must not be longer than 100 characters, or they will be truncated.

Ignore users (donttouchusers)

A list of users to ignore when updating or syncing profiles. Accepts comma-separated usernames.

admin,test,manager,teacher1,teacher2

Enable automatic removal from managed cohorts (enableunenrol)

If this setting is checked, users will also be removed from automatic cohorts that they no longer appear to be a part of, when their profiles are updated or a sync occurs.

Usage example

Say you have custom profile fields "status" (student, teacher or admin) and "classcode", and you want to enrol users into cohorts using both of them.

You could set your main template to:

{{ profile_field_classcode }} - {{ profile_field_status }}s

And empty field text to: none

• A Y19A teacher would be added to a "Y19A - teachers" cohort
• A Y19B student would be added to a "Y19B - students" cohort
• An admin user with no classcode would be added to "none - admins"

To rename the "none - admins" cohort to something more readable, you could add a value to the replace array field:

none - admins|Administrators

When an admin user syncs, they will now be enrolled in a cohort named "Administrators" instead.

moodle-local_cohortauto's Issues

Moodle 4.0 or 4.1

Hi!
Is there a roadmap for 4.x compatibility or it's still not on the menu?
In case you need dev help or help testing functionality let me know!
Filippo

Massive Server Load

Since upgrading to moodle 4, we are experiencing extreme server loads from Cohort_Auto. Is there a way we can make an adjustment to reduce server the server load?

It was set to 15 minutes but takes 12-13 minutes to complete so the server is not recovering, we have changed it to 30 minutes but wondered if we can make any other changes. For example, is changing it to 60 minutes ok? Can we do anything to reduce the server load on the 12 mins run? Does it support Redis?

NULL value for column idnumber is not allowed on Moodle 4.1+ anymore

https://tracker.moodle.org/browse/MDL-77058
https://moodle.org/mod/forum/discuss.php?d=443384

From Moodle 4.1+ onwards, the idnumber column needs to be "string" type. Cohorts created by the plugin has the value NULL for idnumber column, which results in error.

sync_users.php not working in moodle 4

Hi, I've set up the plug in and it is working great for new accounts but when I run sync_users.php in terminal I get the following error:

!!! error/generalexceptionmessage !!!

I'm using moodle 4 and php 7.4

Any ideas?

Translations are not available in AMOS

Hi,
This plugin can not be translated in AMOS.
String concatenation is not allowed.
Please check the plugins checklist for strings at https://docs.moodle.org/dev/index.php?title=Plugin_contribution_checklist&redirect=no#Strings .

Thanks in advance.

UI checks making unit tests failure

After user is created from
https://github.com/moodle/moodle/blob/MOODLE_311_STABLE/lib/tests/report_helper_test.php#L87
it triggers event
https://github.com/catalyst/moodle-local_cohortauto/blob/main/classes/observer.php#L40
in between output is wiped out and $output returns an empty string
https://github.com/moodle/moodle/blob/MOODLE_311_STABLE/lib/tests/report_helper_test.php#L100

And makes this unit test fail:

vendor/bin/phpunit --filter report_helper_test
Moodle 3.11.8+ (Build: 20220722), 417e0252f4e39b28c02d069f35ab5055c8ce39db
Php: 7.4.30, mysqli: 5.7.39, OS: Linux 5.15.0-41-generic x86_64
PHPUnit 9.5.13 by Sebastian Bergmann and contributors.

..F                                                                 3 / 3 (100%)

Time: 00:02.022, Memory: 328.00 MB

There was 1 failure:

1) core\report_helper_test::test_print_report_selector
Failed asserting that '' contains "<option value="/report/log/index.php?id=118000" selected>Logs</option>".

/var/www/cpa-311/lib/tests/report_helper_test.php:107
/var/www/cpa-311/lib/phpunit/classes/advanced_testcase.php:80
phpvfscomposer:///var/www/cpa-311/vendor/phpunit/phpunit/phpunit:97

FAILURES!
Tests: 3, Assertions: 5, Failures: 1.

Is there a way to use email.rootdomain to combine cohorts?

For example lets say I have 3 cohorts employee, partners and customers.
I want all abcCompany.com people to go into the employee cohort.
I can specify the different partner domains to direct all to the partners cohort
But then I want all others not listed above to go into a customer cohort.

Is this possible?

Moodle 3.9 issues

Hello

We have migrated from 3.7 to 3.9 and the plugin has failed to generate any of the cohorts as it did before.

We don't get anything created.
Use old AUTH_MCAE - User logs in - creates the cohort
Convert is using the tool to Auto-Cohort,
Log out learner and log in again, user remains in cohort, doesnt create a new version of the AUTH_MCAE cohort

Kindest Regards
John

Problems when using the Plugin

Do not know where to post this elsewhere but I need some support with the plugin.

I have tried using this with OpenID Connect but it does not seem to work. Does the plugin listen to every login as well or only if a profile gets created/updated?

We have the following superstructure in the company and want to put users into cohorts based on the domain:

• internal users have [email protected]
• external users have [email protected]
• and partner users have [email protected]

So in my case, if I understand how the plugin works correctly I would write I would write
{{ email.domain) }} in the (mainrule_fld) right
and then in (replace_arr)
I would say okay:
companyname.com|Internal
external.companyname.com|Partner
partner.companyname.com|External

Would that make sense?

However, I cannot even find where to modify the plugin after the initial installation. Have not worked with Moodle as much yet so sorry for the inconvenience.

allow match on cohortid

In some cases it would be useful to allow the admin to modify the cohort name after it is created - we should look at using the idnumber field for matching the data so that the display name of the cohort can change.

Support for 3.11?

Will there be supporting Moodle 3.11

Force cohorts to update without logging in users

From what I can understand, this plugin updates cohorts when a user logs in (for that user). Is there any way that we can run a one-off task that performs a bulk sync of all users? We have set-up 13 templates that we'd like to use to sync cohorts, and we have a couple thousand users, but we'd like users to show up in a course gradebook even if they haven't logged in (right now they won't show up until they login because they aren't yet in the cohort). Any ideas?

Can't delete generated cohorts

Hi,
first of all thank you for the development of this useful plugin.

I'm not able to modify (add or remove users) or delete an auto-generated cohort. Is there a way to achieve this? Is this normal?

Thank you

"$PAGE->context was not set" when performing OAuth2 login

After installing this plugin, I'm getting an error when I try to login with an OAuth2 account. I'm not sure if this plugin is causing the error, but it is referenced in the error output:

Coding problem: $PAGE->context was not set. You may have forgotten to call require_login() or $PAGE->set_context(). The page may not display correctly as a result
line 503 of /lib/pagelib.php: call to debugging()
line 830 of /lib/pagelib.php: call to moodle_page->magic_get_context()
line 1456 of /lib/weblib.php: call to moodle_page->__get()
line 214 of /local/cohortauto/lib.php: call to format_string()
line 60 of /local/cohortauto/classes/observer.php: call to local_cohortauto_handler->user_profile_hook()
line ? of unknownfile: call to local_cohortauto_observer::user_updated()
line 155 of /lib/classes/event/manager.php: call to call_user_func()
line 75 of /lib/classes/event/manager.php: call to core\event\manager::process_buffers()
line 834 of /lib/classes/event/base.php: call to core\event\manager::dispatch()
line 209 of /user/lib.php: call to core\event\base->trigger()
line 352 of /auth/oauth2/classes/auth.php: call to user_update_user()
line 480 of /auth/oauth2/classes/auth.php: call to auth_oauth2\auth->update_user()
line 49 of /auth/oauth2/login.php: call to auth_oauth2\auth->complete_login()

I get this error message immediately after logging in, however, I can click 'Continue' and access the site normally.

The URL of the page this error appears on is: /auth/oauth2/login.php?wantsurl=...&sesskey=...&id=...&oauth2code=... (I've removed the actual values of the query string)

Looking for a way to create cohort based on user creation time

Hi,

I guess the field to reference is {{ timecreated }}, but then I'm not entirely sure how.

What is required is to create cohorts based on Month or bi-min they were created e.g.:

• Oct 2020, Nov 2020, Dec 2020
  or
• Sept-Oct 2020, Nov-Dec 2002

etc

Some pointesr would be awsome

50 unicode symbols is too short

It is very strange behavior to cut profile fields. %split function suggests long lists in user's profile, so it is doubtfully.

moodle-local_cohortauto/lib.php

Line 53 in f598df6

$newdata[$key] = substr($str, 0, 100);

settings.php: curious why this plugin is not listed in the local plugin settings

Even though it has a settings.php, it is curious that the link to the settings page for the plugin is not listed on the local plugins sections. Instead, I have to look for "cohortauto" on the search box and then I can see the other two admin pages and also de settings page, with the correct breadcrumb.

I'll try to look into it and try to provide a patch.

Deprecated: Required parameter $result follows optional parameter $prefix in

With Moodle 4 on Apache, MySQL, PHP 8, debug turned on, the following message displayed Deprecated: Required parameter $result follows optional parameter $prefix in /local/cohortauto/lib.php on line 80

support for split+replace and fields on inner arrays on split commands.

Use cases to deal with this issue:

1. As a user with a multivalue custom profile field, separated with a certain delimiter, the user is enrolled to a cohort with a name based on a part of the string of each value.
2. As a user with a custom profile field that is only seen by admin, the value exists only on fields profile.name_custom_profile, and not visible into profile_field_name_custom_profile.

These use cases are necessary to deal with more flexible all profile fields in our case.

We will contribut with a PR in short.

Compatible moodle 4.X ?

Hello teams,

Do you think this plugin will be compatible with moodle 4.X ? Will you make update for the next release ?

Thank you in advance,
Best regards :)

Creating 2 cohorts for most templates

Most, if not all, of our auto-generated cohorts are duplicated with some but not complete overlap in membership.

In some cases, I could see it if there is a multi-value field. But I'm not sure that is the case. What are the reasons cohorts would be duplicated?

warning for $PAGE->context not set when logging in user

In our institution we are using SAML2 plugin for SSO logins.

In conjunction with that, I have reached to see this warning:

Coding problem: $PAGE->context was not set. You may have forgotten to call require_login() or $PAGE->set_context(). The page may not display correctly as a result
line 572 of /lib/pagelib.php: call to debugging()
line 967 of /lib/pagelib.php: call to moodle_page->magic_get_context()
line 1550 of /lib/weblib.php: call to moodle_page->__get()
line 209 of /local/cohortauto/lib.php: call to format_string()
line 60 of /local/cohortauto/classes/observer.php: call to local_cohortauto_handler->user_profile_hook()
line ? of unknownfile: call to local_cohortauto_observer::user_updated()
line 155 of /lib/classes/event/manager.php: call to call_user_func()
line 75 of /lib/classes/event/manager.php: call to core\event\manager::process_buffers()
line 835 of /lib/classes/event/base.php: call to core\event\manager::dispatch()
line 218 of /user/lib.php: call to core\event\base->trigger()
line 1055 of /auth/saml2/classes/auth.php: call to user_update_user()
line 812 of /auth/saml2/classes/auth.php: call to auth_saml2\auth->update_user_profile_fields()
line 635 of /auth/saml2/classes/auth.php: call to auth_saml2\auth->saml_login_complete()
line 43 of /auth/saml2/login.php: call to auth_saml2\auth->saml_login()

By itself, I cannot reach to know how to solve it.

However, looking at the process, it finally invokes the "strip_tags" function. And in the same line I have seen some of the last changes on this plugin. I would suggest to invoke also strip_tags instead of format_string on the lib.php to build the cohort name.

I will try to build a patch for it tomorrow.

Prevent warnings on empty emails

Hi,

This is to ask you suport and consider in your plugin that empty email may exist.

Rationale

In our institution we run user sincronization from Moodle against several sources. Not all of these systems are able to require email as mandatory for users. So, it is probable that some cases passes through the systems and arrives into Moodle with user accounts without email.

We are making every now and then pedagogy of the use of those systems to set up always email addresses.

But, it is difficult to remove this problem.

What we face

We love this plugin and commits to the task we ask it perfectly. However, the cli/sync_users.php process throws a lot of warnings related to empty email addresses, from its lib.php, when consider de domain name of the email address.

We see these lines for every single user that has no email:

PHP Warning:  Undefined array key 1 in /var/www/html/local/cohortauto/lib.php on line 218

Warning: Undefined array key 1 in /var/www/html/local/cohortauto/lib.php on line 218
PHP Deprecated:  explode(): Passing null to parameter #2 ($string) of type string is deprecated in /var/www/html/local/cohortauto/lib.php on line 221

Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated in /var/www/html/local/cohortauto/lib.php on line 221

What we propose

Consider that empty email may happen and proceed the same way, but with empty strings in all those cases.

In particular, we have this patch in our institution:

diff --git a/local/cohortauto/lib.php b/local/cohortauto/lib.php
index 166b7fe4b91..45b46fc794c 100644
--- a/local/cohortauto/lib.php
+++ b/local/cohortauto/lib.php
@@ -214,23 +214,32 @@ class local_cohortauto_handler {
         profile_load_custom_fields($user);
         $userprofiledata = cohortauto_prepare_profile_data($user, $this->config->secondrule_fld);
 
-        // Additional values for email.
-        list($emailusername, $emaildomain) = explode("@", $userprofiledata['email']);
-
-        // Email root domain.
-        $emaildomainarray = explode('.', $emaildomain);
-        if (count($emaildomainarray) > 2) {
-            $emailrootdomain = $emaildomainarray[count($emaildomainarray) - 2].'.'.
-                               $emaildomainarray[count($emaildomainarray) - 1];
+        if (empty($userprofiledata['email'])) {
+            $userprofiledata['email'] = array(
+                'full' => '',
+                'username' => '',
+                'domain' => '',
+                'rootdomain' => '',
+            );
         } else {
-            $emailrootdomain = $emaildomain;
+            // Additional values for email.
+            list($emailusername, $emaildomain) = explode("@", $userprofiledata['email']);
+
+            // Email root domain.
+            $emaildomainarray = explode('.', $emaildomain);
+            if (count($emaildomainarray) > 2) {
+                $emailrootdomain = $emaildomainarray[count($emaildomainarray) - 2] . '.' .
+                    $emaildomainarray[count($emaildomainarray) - 1];
+            } else {
+                $emailrootdomain = $emaildomain;
+            }
+            $userprofiledata['email'] = array(
+                'full' => $userprofiledata['email'],
+                'username' => $emailusername,
+                'domain' => $emaildomain,
+                'rootdomain' => $emailrootdomain
+            );
         }
-        $userprofiledata['email'] = array(
-            'full' => $userprofiledata['email'],
-            'username' => $emailusername,
-            'domain' => $emaildomain,
-            'rootdomain' => $emailrootdomain
-        );
 
         // Set delimiter in use.
         $delimiter = $this->config->delim;

The result

We have applied locally this patch and works like a charm:

# time php local/cohortauto/cli/sync_users.php 
Beginning user cohort sync...
Sync for 79055 users finished.

real    6m52.500s
user    4m36.288s
sys     0m10.032s

And it does not present any warning about empty email.

If you consider this, we are pleased to provide a PR with it.

Let us know.