MX Calibrate
Calibrate a translation table from a set of powder diffraction images taken at various sample-detector distances.
This is a notebook replacement of the MX-calibrate tool from pyFAI with advanced features.
Start with some constant definition:
[1]:
calibrant_name = "CeO2"
detector_name = "Pilatus 2M"
file_pattern = "massif1/test-powder*.cbf"
result_file = "MX-calibrate.json"
wavelength = None # set a value to override the one in the headers
[2]:
!wget http://www.silx.org/pub/pyFAI/massif1.tar.bz2
!tar -xvjf massif1.tar.bz2
--2023-09-07 15:27:28-- http://www.silx.org/pub/pyFAI/massif1.tar.bz2
Résolution de islay (islay)… 192.168.4.1
Connexion à islay (islay)|192.168.4.1|:3128… connecté.
requête Proxy transmise, en attente de la réponse… 200 OK
Taille : 6784503 (6,5M) [application/x-bzip2]
Sauvegarde en : « massif1.tar.bz2.4 »
massif1.tar.bz2.4 100%[===================>] 6,47M --.-KB/s ds 0,1s
2023-09-07 15:27:28 (47,1 MB/s) — « massif1.tar.bz2.4 » sauvegardé [6784503/6784503]
massif1/
massif1/test-powder_5_0001.poni
massif1/test-powder_4_0001.cbf
massif1/test-powder_6_0001.poni
massif1/test-powder_8_0001.cbf
massif1/test-powder_7_0001.cbf
massif1/test-powder_7_0001.poni
massif1/test-powder_3_0001.cbf
massif1/test-powder_1_0001.poni
massif1/test-powder_4_0001.poni
massif1/test-powder_8_0001.poni
massif1/test-powder_2_0001.poni
massif1/test-powder_6_0001.cbf
massif1/test-powder_2_0001.cbf
massif1/test-powder_5_0001.cbf
massif1/test-powder_3_0001.poni
massif1/test-powder_1_0001.cbf
[3]:
%matplotlib widget
#inline
[4]:
import os
import glob
import logging
import numpy
from matplotlib.pyplot import subplots
import fabio
import pyFAI
from pyFAI.gui import jupyter
import pyFAI.calibrant
from pyFAI.gui.jupyter.calib import Calibration
from pyFAI.goniometer import GeometryTransformation, GoniometerRefinement
from pyFAI.gui.cli_calibration import AbstractCalibration
import pyFAI.gui.mpl_calib
pyFAI.gui.mpl_calib.logger.setLevel(logging.ERROR)
WARNING:pyFAI.gui.matplotlib:matplotlib already loaded, setting its backend may not work
/home/jerome/.venv/py311/lib/python3.11/site-packages/pyopencl/cache.py:495: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
_create_built_program_from_source_cached(
/home/jerome/.venv/py311/lib/python3.11/site-packages/pyopencl/cache.py:499: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
prg.build(options_bytes, devices)
[5]:
detector = pyFAI.detector_factory(detector_name)
calibrant = pyFAI.calibrant.get_calibrant(calibrant_name)
files = sorted(glob.glob(file_pattern))
print("input files: "+" ".join(files))
input files: massif1/test-powder_1_0001.cbf massif1/test-powder_2_0001.cbf massif1/test-powder_3_0001.cbf massif1/test-powder_4_0001.cbf massif1/test-powder_5_0001.cbf massif1/test-powder_6_0001.cbf massif1/test-powder_7_0001.cbf massif1/test-powder_8_0001.cbf
[6]:
first = fabio.open(files[0])
def get_dectris_headers(fimg):
"""return the dectris headers from a Pilatus detector"""
res = {}
for line in fimg.header.get("_array_data.header_contents", "").split("\n"):
words = line.split()
if len(words)>=3:
key = words[1]
for v in words[2:]:
try:
vf = float(v)
except:
continue
if not("." in v or "e" in v):
vf = int(v)
res[key] = vf
return res
get_dectris_headers(first)
[6]:
{'Silicon': 0.00045,
'Pixel_size': 0.000172,
'N_oscillations': 1,
'Chi': 0.0,
'Phi': 0.0,
'Kappa': 0.0,
'Alpha': 0.0,
'Polarization': 0.99,
'Detector_2theta': 0.0,
'Angle_increment': 1.0,
'Transmission': 100.0,
'Flux': 436215830143.2828,
'Detector_Voffset': 0.0,
'Detector_distance': 0.126474,
'Wavelength': 0.965459,
'N_excluded_pixels:': 321,
'Threshold_setting': 6421,
'Count_cutoff': 1048500,
'Tau': 0,
'Exposure_period': 0.02115,
'Exposure_time': 0.02,
'Start_angle': 0.0}
[7]:
if wavelength is None:
wavelength = get_dectris_headers(first)["Wavelength"] * 1e-10
calibrant.wavelength = wavelength
[8]:
#apply mask to the detector
mask = numpy.logical_or(detector.mask, first.data<0)
detector.mask = mask
Manual calibration of the first image
[9]:
# Important: select the ring number before right-click on the ring. Finally click on the refine button
calib = Calibration(img=first.data,
mask=mask,
detector=detector,
wavelength=wavelength,
calibrant=calibrant)
[10]:
calib.extract_cpt()
# calib.geoRef.rot1 = calib.geoRef.rot2 = calib.geoRef.rot3 = 0
calib.refine(fixed=["wavelength", "rot3"])
Before refinement, the geometry is:
Detector Pilatus 2M PixelSize= 1.720e-04, 1.720e-04 m
Wavelength= 9.654590e-11 m
SampleDetDist= 1.261254e-01 m PONI= 1.483899e-01, 1.219401e-01 m rot1=0.011442 rot2=-0.012514 rot3=0.000000 rad
DirectBeamDist= 126.144 mm Center: x=700.563, y=853.554 pix Tilt= 0.972° tiltPlanRotation= -132.436° 𝛌= 0.965Å
Detector Pilatus 2M PixelSize= 1.720e-04, 1.720e-04 m
Wavelength= 9.654590e-11 m
SampleDetDist= 1.263011e-01 m PONI= 1.474121e-01, 1.212166e-01 m rot1=0.006290 rot2=-0.005380 rot3=0.000000 rad
DirectBeamDist= 126.305 mm Center: x=700.129, y=853.096 pix Tilt= 0.474° tiltPlanRotation= -139.456° 𝛌= 0.965Å
Detector Pilatus 2M PixelSize= 1.720e-04, 1.720e-04 m
Wavelength= 9.654590e-11 m
SampleDetDist= 1.263011e-01 m PONI= 1.474121e-01, 1.212166e-01 m rot1=0.006290 rot2=-0.005380 rot3=0.000000 rad
DirectBeamDist= 126.305 mm Center: x=700.129, y=853.096 pix Tilt= 0.474° tiltPlanRotation= -139.456° 𝛌= 0.965Å
Detector Pilatus 2M PixelSize= 1.720e-04, 1.720e-04 m
Wavelength= 9.654590e-11 m
SampleDetDist= 1.263011e-01 m PONI= 1.474121e-01, 1.212166e-01 m rot1=0.006290 rot2=-0.005380 rot3=0.000000 rad
DirectBeamDist= 126.305 mm Center: x=700.129, y=853.096 pix Tilt= 0.474° tiltPlanRotation= -139.456° 𝛌= 0.965Å
[11]:
#Return to `inline` mode
%matplotlib inline
calib.peakPicker.widget.fig.show()
Check that the beam-center and the distance is correct and how much they are off.
Calibration of all frames in automatic mode:
[12]:
# Definition of the geometry translation function:
get_distance = lambda fimg: get_dectris_headers(fimg)["Detector_distance"]
geotrans = GeometryTransformation(param_names = ["dist_offset",
"poni1", "poni2", "rot1","rot2",
"dist_scale", "poni1_scale", "poni2_scale"],
dist_expr="pos * dist_scale + dist_offset",
poni1_expr="pos * poni1_scale + poni1",
poni2_expr="pos * poni2_scale + poni2",
rot1_expr="rot1",
rot2_expr="rot2",
rot3_expr="0.0")
param = {
"dist_offset": calib.geoRef.dist-get_distance(first),
"poni1": calib.geoRef.poni1,
"poni2": calib.geoRef.poni2,
"rot1": calib.geoRef.rot1,
"rot2": calib.geoRef.rot2,
"dist_scale": 1.0,
"poni1_scale": 0.0,
"poni2_scale": 0.0,
}
print(param)
{'dist_offset': -0.00017293552155364655, 'poni1': 0.14741208731439442, 'poni2': 0.12121661313764838, 'rot1': 0.006290118213917636, 'rot2': -0.005380493882842351, 'dist_scale': 1.0, 'poni1_scale': 0.0, 'poni2_scale': 0.0}
[13]:
# Definition of the geometry refinement: the parameter order is the same as the param_names
gonioref = GoniometerRefinement(param, #initial guess
pos_function=get_distance,
trans_function=geotrans,
detector=detector,
wavelength=wavelength)
print("Empty refinement object:")
print(gonioref)
Empty refinement object:
GoniometerRefinement with 0 geometries labeled: .
[14]:
# Let's populate the goniometer refinement object with all geometries:
for fn in files:
base = os.path.splitext(fn)[0]
fimg = fabio.open(fn)
local_calib = AbstractCalibration(img=fimg.data, mask=mask,
detector=detector,
wavelength=wavelength,
calibrant=calibrant)
local_calib.preprocess()
local_calib.fixed = ["wavelength", "rot3"]
local_calib.ai = gonioref.get_ai(get_distance(fimg))
local_calib.extract_cpt()
sg = gonioref.new_geometry(os.path.basename(base), image=fimg.data, metadata=fimg,
control_points=local_calib.peakPicker.points,
geometry=local_calib.ai,
calibrant=calibrant)
print("Filled refinement object:")
print(gonioref)
print(os.linesep+"\tLabel \t Distance")
for k, v in gonioref.single_geometries.items():
print(k,v.get_position())
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
ERROR:root:No diffraction image available => not showing the contour
Filled refinement object:
GoniometerRefinement with 8 geometries labeled: test-powder_1_0001, test-powder_2_0001, test-powder_3_0001, test-powder_4_0001, test-powder_5_0001, test-powder_6_0001, test-powder_7_0001, test-powder_8_0001.
Label Distance
test-powder_1_0001 0.126474
test-powder_2_0001 0.141749
test-powder_3_0001 0.199249
test-powder_4_0001 0.171074
test-powder_5_0001 0.226674
test-powder_6_0001 0.293162
test-powder_7_0001 0.357899
test-powder_8_0001 0.484611
[15]:
fig, ax = subplots(len(files), figsize=(10, 10*len(files)))
for sp, sg in zip(ax, gonioref.single_geometries.values()):
jupyter.display(sg=sg, ax=sp, label=sg.label)
[16]:
gonioref.refine3(fix=["dist_scale", "poni1_scale", "poni2_scale"])
Free parameters: ['dist_offset', 'poni1', 'poni2', 'rot1', 'rot2']
Fixed: {'dist_scale': 1.0, 'poni1_scale': 0.0, 'poni2_scale': 0.0}
message: Optimization terminated successfully
success: True
status: 0
fun: 3.842311156114474e-07
x: [-9.369e-05 1.471e-01 1.209e-01 4.167e-03 -2.916e-03]
nit: 12
jac: [ 1.478e-08 -1.613e-08 2.289e-08 -1.154e-08 -7.511e-09]
nfev: 76
njev: 12
Constrained Least square 1.143651203143249e-06 --> 3.842311156114474e-07
maxdelta on rot2: -0.005380493882842351 --> -0.0029157457711422004
[16]:
3.842311156114474e-07
[17]:
gonioref.refine3(fix=[])
Free parameters: ['dist_offset', 'poni1', 'poni2', 'rot1', 'rot2', 'dist_scale', 'poni1_scale', 'poni2_scale']
Fixed: {}
message: Optimization terminated successfully
success: True
status: 0
fun: 3.1666683348137045e-07
x: [-3.471e-04 1.470e-01 1.208e-01 5.616e-03 -3.268e-03
1.002e+00 1.081e-03 2.606e-03]
nit: 23
jac: [ 8.122e-08 1.221e-07 -1.483e-07 -8.454e-08 -8.413e-08
2.620e-08 2.569e-07 -6.400e-08]
nfev: 209
njev: 23
Constrained Least square 3.842311156114474e-07 --> 3.1666683348137045e-07
maxdelta on poni2_scale: 0.0 --> 0.0026060466587728313
[17]:
3.1666683348137045e-07
Interpretation of this fit:
[18]:
gonioref.get_ai(0.2)
[18]:
Detector Pilatus 2M PixelSize= 1.720e-04, 1.720e-04 m
Wavelength= 9.654590e-11 m
SampleDetDist= 1.999799e-01 m PONI= 1.471968e-01, 1.213406e-01 m rot1=0.005616 rot2=-0.003268 rot3=0.000000 rad
DirectBeamDist= 199.984 mm Center: x=698.938, y=851.995 pix Tilt= 0.372° tiltPlanRotation= -149.804° 𝛌= 0.965Å
[19]:
gonioref.get_ai(0.3)
[19]:
Detector Pilatus 2M PixelSize= 1.720e-04, 1.720e-04 m
Wavelength= 9.654590e-11 m
SampleDetDist= 3.001434e-01 m PONI= 1.473049e-01, 1.216012e-01 m rot1=0.005616 rot2=-0.003268 rot3=0.000000 rad
DirectBeamDist= 300.150 mm Center: x=697.183, y=850.720 pix Tilt= 0.372° tiltPlanRotation= -149.804° 𝛌= 0.965Å
Persistence of this fit
[20]:
gonioref.save(result_file)
[21]:
with open(result_file) as r:
print(r.read())
{
"content": "Goniometer calibration v2",
"detector": "Pilatus 2M",
"detector_config": {},
"wavelength": 9.65459e-11,
"param": [
-0.00034712406037136305,
0.14698057683457763,
0.12081938334221404,
0.005616434336593687,
-0.003268310371393127,
1.001635041769471,
0.0010810657845790855,
0.0026060466587728313
],
"param_names": [
"dist_offset",
"poni1",
"poni2",
"rot1",
"rot2",
"dist_scale",
"poni1_scale",
"poni2_scale"
],
"pos_names": [
"pos"
],
"trans_function": {
"content": "GeometryTransformation",
"param_names": [
"dist_offset",
"poni1",
"poni2",
"rot1",
"rot2",
"dist_scale",
"poni1_scale",
"poni2_scale"
],
"pos_names": [
"pos"
],
"dist_expr": "pos * dist_scale + dist_offset",
"poni1_expr": "pos * poni1_scale + poni1",
"poni2_expr": "pos * poni2_scale + poni2",
"rot1_expr": "rot1",
"rot2_expr": "rot2",
"rot3_expr": "0.0",
"constants": {
"pi": 3.141592653589793
}
}
}
Interpretation of the fit:
[22]:
distances = []
f_distances = []
f_poni1 = []
f_poni2 = []
g_distances = []
g_poni1 = []
g_poni2 = []
for sg in gonioref.single_geometries.values():
distance = sg.get_position()
distances.append(distance)
sg.geometry_refinement.refine3(fix=["wavelength", "rot3"])
f_distances.append(sg.geometry_refinement.dist)
f_poni1.append(sg.geometry_refinement.poni1)
f_poni2.append(sg.geometry_refinement.poni2)
ai = gonioref.get_ai(distance)
g_distances.append(ai.dist)
g_poni1.append(ai.poni1)
g_poni2.append(ai.poni2)
[23]:
fig,ax = subplots(2)
ax[0].plot(distances, f_distances, label="fitted individually")
ax[0].plot(distances, g_distances, label="fitted table")
ax[0].set_title("Observed deviations:")
ax[1].set_xlabel("Encoder distance (m)")
ax[1].plot(distances, f_poni1, label="poni1 fitted individually")
ax[1].plot(distances, g_poni1, label="poni1 fitted table")
ax[1].plot(distances, f_poni2, label="poni2 fitted individually")
ax[1].plot(distances, g_poni2, label="poni2 fitted table")
ax[0].set_ylabel("Fitted distance (m)")
ax[1].set_ylabel("Fitted PONIs (m)")
ax[0].legend()
ax[1].legend()
pass
Nota:
The degradation between 0.3 and 0.5m correspond to the image 6->7 and is related to the disparition of the third ring!
Conclusion:
This notebook demonstrates: * The usage of the geometry calibration in Jupyter-lab to calibrate the first image * The creation of a goniometer-refinement * The population of this goniometer-refinement with automatic control-point extraction * The fit of the table, first with the constrains of a perfecty aligned table, then with a mis-aligned table
In our case the table is miss-aligned in the horizontal direction by 2.3mm/meter (i.e. 2.3 mradian). This should be taken into account when calculating the beam-center at different distances.