Why the evaluation on test using the sample_clients from training in the EMNIST tutorial? about federated

[ZacharyGarrett]

This is by design; from the documentation for tff.simulation.datasets.emnist.load_data():

Rather than holding out specific users, each user's examples are split across train and test so that all users have at least one example in train and one example in test. Writers that had less than 2 examples are excluded from the data set.

So the while the evaluation is processing the same clients, it is not processing the same examples.

As you suggest, holding out entire clients (instead of examples) definitely seems reasonable and is an alternative way to evaluate the quality of a model.