Note
Click here to download the full example code
07. Command-line interface#
The implemented command-line interface (CLI) can be powerful to use emg3d from any programming environment. It enables, for instance, to use emg3d as a forward modelling kernel in your inversion code running on a server. The capabilities of the CLI are more limited than of the Python interface, but they are sufficient to compute the responses of a given survey for a provided model, and to compute the gradient of the misfit function.
The biggest difficulty to work with the CLI is, at the moment, the file formats, or better the lack of their documentation. Surveys, models, and grids can be stored with emg3d to either h5, npz, or json files. The status of the documentation will improve towards the v1.0.0 release of emg3d. The “easiest” way at the moment is to generate such a file in Python, and reproduce its structure in your language or program of choice. Please feel free to open an issue on GitHub if there are questions in this regard. The idea for the future is to go through https://github.com/softwareunderground/subsurface to interface to various file formats.
import os
import pooch
import subprocess
# Adjust this path to a folder of your choice.
data_path = os.path.join('..', 'download', '')
Note that everything shown in this example is meant to be executed in a
terminal, nothing is executed in Python. However, as this example gallery is
generated in Python we have to use a work-around to show the functionality,
the function bash(command). You would execute the provided command in
your terminal.
def bash(command):
"Prints the `command`, executes it in bash, and prints the output."""
# Print command
print(f"$ {command}")
# Move to data_path
command = f"cd {data_path}; " + command
# Carry out command
msg = subprocess.run(command, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
# Print output
print(msg.stdout.decode())
We have to fetch the example files. You can also download these manually or provide your own survey, model, and parameter files.
model = "GemPy-II.h5"
pooch.retrieve(
'https://raw.github.com/emsig/data/2021-05-21/emg3d/models/'+model,
'ea8c23be80522d3ca8f36742c93758370df89188816f50cb4e1b2a6a3012d659',
fname=model,
path=data_path,
)
survey = 'GemPy-II-survey-A.h5'
pooch.retrieve(
'https://raw.github.com/emsig/data/2021-05-21/emg3d/surveys/'+survey,
'5f2ed0b959a4f80f5378a071e6f729c6b7446898be7689ddc9bbd100a8f5bce7',
fname=survey,
path=data_path,
)
'/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II-survey-A.h5'
Getting started - help#
The best way to get started is, as with any bash command, to have a look at its in-built help.
bash("emg3d --help")
$ emg3d --help
usage: emg3d [-h] [-n NPROC] [-f | -m | -g] [--path PATH] [--survey SURVEY]
[--model MODEL] [--output OUTPUT] [--save SAVE] [--load LOAD]
[--cache CACHE] [--clean] [-l] [-d]
[--verbosity {-1,0,1,2} | -v | -q] [--report] [--version]
[config]
Multigrid solver for 3D electromagnetic diffusion.
positional arguments:
config name of config file; default is 'emg3d.cfg'; consult
https://emg3d.emsig.xyz/en/stable/manual/cli.html for
its format
optional arguments:
-h, --help show this help message and exit
-n NPROC, --nproc NPROC
number of processors
-f, --forward compute synthetic data (default)
-m, --misfit compute synthetic data and their misfit
-g, --gradient compute synthetic data, misfit, and its gradient
--path PATH path (abs or rel); file names are relative to path
--survey SURVEY input survey file name; default is 'survey.h5'
--model MODEL input model file name; default is 'model.h5'
--output OUTPUT output files base name; default is 'emg3d_out'
--save SAVE file-name to save simulation; not used by default
--load LOAD file-name to load simulation; not used by default; if
set, the provided survey and almost all parameters in
the config file are ignored; provided model is only
used if --clean is set
--cache CACHE shortcut: file-name for --load --save (cache overrules
load and save)
--clean replace model and all computed data of loaded
simulation
-l, --layered use a layered model with empymod for each src-rec pair
-d, --dry-run only display what would have been done
--verbosity {-1,0,1,2}
set verbosity; default is 0
-v, --verbose increase verbosity; can be used multiple times
-q, --quiet decrease verbosity
--report only display emg3d report
--version only display emg3d version
Configuration file#
The CLI is driven by a configuration file, that is by default named
emg3d.cfg and must be in the same directory as you execute the command.
If it has a different name or a different location you have to provide the
full or relative path and filename as the first argument to emg3d.
The configuration parameters are described in the documentation, consult Manual -> CLI.
Let’s write a simple example file.
with open(f"{data_path}emg3d.cfg", "w") as f:
f.write("""[files]
survey = GemPy-II-survey-A.h5
model = GemPy-II.h5
output = emg3d_out.json
[simulation]
name = CLI Test
gridding = single
[solver_opts]
# nothing specified
[gridding_opts]
center_on_edge = False
[data]
sources = TxED-1
receivers = RxEP-08, RxEP-38
frequencies = f-1""")
The file defines the location of the survey and model files, the name and format of the output file, and selects a source, two receivers, and a frequency to compute.
Compute the forward responses#
We just compute the forward response here, but the usage for the misfit or the gradient is very similar. We use the most verbose version here, to see what it does internally.
bash("emg3d --forward -vv")
$ emg3d --forward -vv
:: emg3d CLI forward START :: Wed Aug 31 21:34:45 2022 :: v1.8.0
--------------------------------------------------------------------------------
Date: Wed Aug 31 21:34:45 2022 CEST
OS : Linux
CPU(s) : 4
Machine : x86_64
Architecture : 64bit
RAM : 15.5 GiB
Environment : Python
File system : ext4
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:22:55)
[GCC 10.3.0]
numpy : 1.22.4
scipy : 1.9.0
numba : 0.55.2
emg3d : 1.8.0
empymod : 2.2.0
xarray : 2022.6.0
discretize : 0.8.2
h5py : 3.7.0
matplotlib : 3.4.3
tqdm : 4.64.0
IPython : 8.4.0
Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112
for Intel(R) 64 architecture applications
--------------------------------------------------------------------------------
:: CONFIGURATION ::
/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d.cfg
{
"data": {
"frequencies": [
"f-1"
],
"receivers": [
"RxEP-08",
"RxEP-38"
],
"sources": [
"TxED-1"
]
},
"files": {
"load": false,
"log": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.log",
"model": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II.h5",
"output": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.json",
"save": false,
"survey": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II-survey-A.h5"
},
"noise_kwargs": {},
"simulation_options": {
"gridding": "single",
"gridding_opts": {
"center_on_edge": false
},
"name": "CLI Test"
}
}
:: LOAD SURVEY AND MODEL ::
Data loaded from «/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II-survey-A.h5»
[emg3d v0.17.1.dev22+ge23c468 (format 1.0) on 2021-04-14T22:24:03.229285].
Data loaded from «/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II.h5»
[emg3d v1.0.0rc3.dev5+g0cd9e09 (format 1.0) on 2021-05-21T18:40:16.721968].
:: SIMULATION ::
:: Simulation «CLI Test» ::
- Survey «GemPy-II Survey A»: 1 sources; 2 receivers; 1 frequencies
- Model: resistivity; isotropic; 100 x 100 x 102 (1,020,000)
- Gridding: Single grid for all sources and frequencies; 128 x 64 x 64 (524,288)
:: MESHES ::
= Source: all; Frequency: all =
== GRIDDING IN X ==
Skin depth [m] : 390 / 7117625 / 7117625 [corr. to `properties`]
Survey dom. DS [m] : 4000 - 16000
Comp. dom. DC [m] : -96000 - 116000
Final extent [m] : -101181 - 121187
Cell widths [m] : 130 / 130 / 28717 [min(DS) / max(DS) / max(DC)]
Number of cells : 128 (94 / 34 / 0) [Total (DS/DC/remain)]
Max stretching : 1.000 (1.000) / 1.374 [DS (seasurface) / DC]
== GRIDDING IN Y ==
Skin depth [m] : 390 / 7117625 / 7117625 [corr. to `properties`]
Survey dom. DS [m] : 5500 - 9500
Comp. dom. DC [m] : -94500 - 109500
Final extent [m] : -97055 - 153667
Cell widths [m] : 130 / 130 / 41613 [min(DS) / max(DS) / max(DC)]
Number of cells : 64 (31 / 33 / 0) [Total (DS/DC/remain)]
Max stretching : 1.000 (1.000) / 1.404 [DS (seasurface) / DC]
== GRIDDING IN Z ==
Skin depth [m] : 390 / 2251 / 7117625 [corr. to `properties`]
Survey dom. DS [m] : -6725 - -1722
Comp. dom. DC [m] : -20867 - 98278
Final extent [m] : -23065 - 102852
Cell widths [m] : 130 / 130 / 32306 [min(DS) / max(DS) / max(DC)]
Number of cells : 64 (39 / 25 / 0) [Total (DS/DC/remain)]
Max stretching : 1.000 (1.000) / 1.444 [DS (seasurface) / DC]
TensorMesh: 524,288 cells
MESH EXTENT CELL WIDTH FACTOR
dir nC min max min max max
--- --- --------------------------- ------------------ ------
x 128 -101,180.95 121,187.06 129.95 28,716.91 1.37
y 64 -97,054.53 153,667.41 129.95 41,612.88 1.40
z 64 -23,064.76 102,852.13 129.95 32,306.29 1.44
:: FORWARD COMPUTATION ::
Compute efields 0/1 [00:00]
Compute efields ██████████ 1/1 [00:42]
Compute efields ██████████ 1/1 [00:42]
- SOLVER INFO <efield> -
= Source TxED-1; Frequency 0.5 Hz = 7.5e-07; 1(5); 0:00:42; CONVERGED
:: SAVE RESULTS ::
Data saved to «/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.json»
[emg3d v1.8.0 (format 1.0) on 2022-08-31T21:35:28.151599].
:: emg3d CLI forward END :: Wed Aug 31 21:35:28 2022 :: runtime = 0:00:43
Generated files#
The run created the files emg3d_out.json (name and file-type can be
changed in the config file), which contains the responses, and the file
emg3d_out.log with some information.
bash("ls emg3d* -1")
$ ls emg3d* -1
emg3d.cfg
emg3d_out.json
emg3d_out.log
Log#
Let’s have a look at the log, which is mostly the same as was printed above with the verbosity flag:
bash("cat emg3d_out.log")
$ cat emg3d_out.log
:: emg3d CLI forward START :: Wed Aug 31 21:34:45 2022 :: v1.8.0
--------------------------------------------------------------------------------
Date: Wed Aug 31 21:34:45 2022 CEST
OS : Linux
CPU(s) : 4
Machine : x86_64
Architecture : 64bit
RAM : 15.5 GiB
Environment : Python
File system : ext4
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:22:55)
[GCC 10.3.0]
numpy : 1.22.4
scipy : 1.9.0
numba : 0.55.2
emg3d : 1.8.0
empymod : 2.2.0
xarray : 2022.6.0
discretize : 0.8.2
h5py : 3.7.0
matplotlib : 3.4.3
tqdm : 4.64.0
IPython : 8.4.0
Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112
for Intel(R) 64 architecture applications
--------------------------------------------------------------------------------
:: CONFIGURATION ::
/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d.cfg
{
"data": {
"frequencies": [
"f-1"
],
"receivers": [
"RxEP-08",
"RxEP-38"
],
"sources": [
"TxED-1"
]
},
"files": {
"load": false,
"log": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.log",
"model": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II.h5",
"output": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.json",
"save": false,
"survey": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II-survey-A.h5"
},
"noise_kwargs": {},
"simulation_options": {
"gridding": "single",
"gridding_opts": {
"center_on_edge": false
},
"name": "CLI Test"
}
}
:: LOAD SURVEY AND MODEL ::
Data loaded from «/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II-survey-A.h5»
[emg3d v0.17.1.dev22+ge23c468 (format 1.0) on 2021-04-14T22:24:03.229285].
Data loaded from «/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II.h5»
[emg3d v1.0.0rc3.dev5+g0cd9e09 (format 1.0) on 2021-05-21T18:40:16.721968].
:: SIMULATION ::
:: Simulation «CLI Test» ::
- Survey «GemPy-II Survey A»: 1 sources; 2 receivers; 1 frequencies
- Model: resistivity; isotropic; 100 x 100 x 102 (1,020,000)
- Gridding: Single grid for all sources and frequencies; 128 x 64 x 64 (524,288)
:: MESHES ::
= Source: all; Frequency: all =
== GRIDDING IN X ==
Skin depth [m] : 390 / 7117625 / 7117625 [corr. to `properties`]
Survey dom. DS [m] : 4000 - 16000
Comp. dom. DC [m] : -96000 - 116000
Final extent [m] : -101181 - 121187
Cell widths [m] : 130 / 130 / 28717 [min(DS) / max(DS) / max(DC)]
Number of cells : 128 (94 / 34 / 0) [Total (DS/DC/remain)]
Max stretching : 1.000 (1.000) / 1.374 [DS (seasurface) / DC]
== GRIDDING IN Y ==
Skin depth [m] : 390 / 7117625 / 7117625 [corr. to `properties`]
Survey dom. DS [m] : 5500 - 9500
Comp. dom. DC [m] : -94500 - 109500
Final extent [m] : -97055 - 153667
Cell widths [m] : 130 / 130 / 41613 [min(DS) / max(DS) / max(DC)]
Number of cells : 64 (31 / 33 / 0) [Total (DS/DC/remain)]
Max stretching : 1.000 (1.000) / 1.404 [DS (seasurface) / DC]
== GRIDDING IN Z ==
Skin depth [m] : 390 / 2251 / 7117625 [corr. to `properties`]
Survey dom. DS [m] : -6725 - -1722
Comp. dom. DC [m] : -20867 - 98278
Final extent [m] : -23065 - 102852
Cell widths [m] : 130 / 130 / 32306 [min(DS) / max(DS) / max(DC)]
Number of cells : 64 (39 / 25 / 0) [Total (DS/DC/remain)]
Max stretching : 1.000 (1.000) / 1.444 [DS (seasurface) / DC]
TensorMesh: 524,288 cells
MESH EXTENT CELL WIDTH FACTOR
dir nC min max min max max
--- --- --------------------------- ------------------ ------
x 128 -101,180.95 121,187.06 129.95 28,716.91 1.37
y 64 -97,054.53 153,667.41 129.95 41,612.88 1.40
z 64 -23,064.76 102,852.13 129.95 32,306.29 1.44
:: FORWARD COMPUTATION ::
- SOLVER INFO <efield> -
= Source TxED-1; Frequency 0.5 Hz = 7.5e-07; 1(5); 0:00:42; CONVERGED
:: SAVE RESULTS ::
Data saved to «/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.json»
[emg3d v1.8.0 (format 1.0) on 2022-08-31T21:35:28.151599].
:: emg3d CLI forward END :: Wed Aug 31 21:35:28 2022 :: runtime = 0:00:43
Result / output#
We stored the output as a json-file, just so we can easily cat it. However,
you might want to use h5 files, as they are compressed. Note that json cannot
store complex numbers, so the responses are stored as [[Re], [Im]].
bash("cat emg3d_out.json")
$ cat emg3d_out.json
{
"configuration": {
"files": {
"save": false,
"load": false,
"survey": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II-survey-A.h5",
"model": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/GemPy-II.h5",
"output": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.json",
"log": "/home/dtr/3-GitHub/emsig/emg3d-gallery/examples/download/emg3d_out.log"
},
"simulation_options": {
"gridding": "single",
"name": "CLI Test",
"gridding_opts": {
"center_on_edge": false
}
},
"data": {
"sources": [
"TxED-1"
],
"receivers": [
"RxEP-08",
"RxEP-38"
],
"frequencies": [
"f-1"
]
},
"noise_kwargs": {}
},
"data__complex__array-float64": [
[
[
[
-3.643887691173835e-08
],
[
NaN
]
]
],
[
[
[
-3.3514219146614606e-09
],
[
NaN
]
]
]
],
"_date": "2022-08-31T21:35:28.151599",
"_version": "emg3d v1.8.0",
"_format": "1.0"
}
Report#
The report-command prints an extensive list of your python environment of components which are important for emg3d. If you report an issue it helps a lot if you provide this info.
bash("emg3d --report")
$ emg3d --report
--------------------------------------------------------------------------------
Date: Wed Aug 31 21:35:31 2022 CEST
OS : Linux
CPU(s) : 4
Machine : x86_64
Architecture : 64bit
RAM : 15.5 GiB
Environment : Python
File system : ext4
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:22:55)
[GCC 10.3.0]
numpy : 1.22.4
scipy : 1.9.0
numba : 0.55.2
emg3d : 1.8.0
empymod : 2.2.0
xarray : 2022.6.0
discretize : 0.8.2
h5py : 3.7.0
matplotlib : 3.4.3
tqdm : 4.64.0
IPython : 8.4.0
Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112
for Intel(R) 64 architecture applications
--------------------------------------------------------------------------------
Total running time of the script: ( 0 minutes 50.098 seconds)
Estimated memory usage: 9 MB