Cloud Compare with Python automation

[benb]

Hi,
I'd like to build a little interface that will load a bunch of LAS files into CC, process them and save out files. Pretty straightforward.
I'd like to control it with Python and use multiple instances of CC running concurrently, with Python feeding files to the instances as they complete their processing of the individual LAS files... If i can get it to work for a single instance of CC, then I'd like to expand it to multiprocessing.

it may be helpful for best assessment to review the specific workflow I hope to achieve, and to determine if these can be accomplished perhaps through the plugin, CLI, or the CloudComPy interface (which I have found VERY difficult to build for use). CloudComPy requires Qt5 which now does not provide some required dependencies, and I have run into days of wasted time trying to build the CloudComPy tool. It seems broken to me. Or maybe it broke me! I have temproarily quit trying to build CloudComPy with CMake and Visual Studio, it was hellish.... anyway my goal is pretty simple and I could do this all by hand but automation by software control seems sensible - so seeking a howto that actually works is my state now.

First, Open the next file.
Then in the Open LAS file window/ Loading/Standard Fields/ Select All/ Apply.
Then in the Global Shift/Scale window, click Yes.
Then in the DB Tree window, select the lowest child element in the DB Tree.
Then in the Properties window, under Scalar Fields (note: NOT under CC Object/ Scalar field !!), set Scalar Fields/ Active to Classification.
Then activate the Min/Max tool (also known as the Filter points by value tool) and set its Range to be 1.1 to 2.1 and select Export.
Then ensure that the newly filtered point cloud (lowest child object in the DBtree is selected.
Then export/save the filtered point cloud with a proper name to a proper lcation.
Then with the filtered point cloud still selected in the DB Tree, activate the Tools/ Projection/ Rasterize (and contour plot) tool (also known as the Convert a cloud to 2D raster tool).
Then, in the Rasterize tool set step to desired setting (let's say 1.0 to get started to save time during testing, we can make an .ini file to make that setting be user-accessible, we will want to set it during production to .5, .3, or other setting), active layer= Cell height values, Projection direction= Z, cell height= median, Std. dev. layer= Intensity, project SF's= ON and median value selected, resample point cloud= ON, Empty cells/ Fill with= let's start with Interpolation for getting started, (we will use kriging during production, this can be user- selectable, like step height, in the .ini file).
Then it should Activate the Update Grid function, which will open the Grid Size window. Select Yes to that window.
Then observe for completion of the updating of the grid.
Then when the grid is updated, in the Export tab of the Rasterize tab, select Raster, name the file properly and set its Save location properly and save it.

Therefore the processing of each file would produce two derivative files; the first would be an LAS file of the filtered point cloud, the second would be a TIF raster of the rasterized filtered point cloud.

Is there anyone who could help me with this in any way? Suggestions, collaboration, recommendations for solutions??
Should I try building CloudComPy with Docker or Conda instead of CMmake/VisualStudio?
I spent time building a batch file but learned it cannot control the tools inside CC, so I need something that can do that. The Plugin maybe?
I thought external control would be most flexible and extensible, but CloudComPy is a grizzly bear to build, so I am open to new ideas.

Thank you,

-benb

[daniel]

So first, CloudCompy is a different project. Any question related to CloudCompy should be asked on its github page. If can be about your compilation issues, or about whether this is feasible or not.

But on my side, I can say that most of these operations can already be performed with the CLI mode. If not all... And if something is missing, it may not be too complicated to simply create or improve the CLI option that's missing or incomplete.

[benb]

I will review your Wiki docs for more information about the CLI.
Would it make sense to manage the CLI with Python scripting?
Simple is good, my task objectives are few and not complex.
I have a collection of LAS files to filter, rasterize, and save.
My goal is to automate to get it done consistently and fast.
There are only about 600 files, each is about 1gb in size.
They take about an hour to "Upgrade grid" to rasterize,
I wonder if there is a way to "listen" to their progress
when "Upgrade grid" process is complete, file save.
It could be done crudely with a timer, but that is
not a secure way to ensure great consistency.
Such are the considerations i am pondering.
CloudComPy requires many dependencies.
I'd like to keep things as simple as I can.
The CLI sounds like a possible solution
if the issue of saving grid is soluble.
An elegant solution might require
some sort of low level trigger
to facilitate the Save event.
I will see the Wiki entries.
Maybe I will post again.
I will check again also
to see any new reply.
Keeping it simple,
getting it done.
fast and right,
Thank you:
-benb

[daniel]

See the -OUTPUT_RASTER_XXX options to export the grid as a geotiff image (see https://www.cloudcompare.org/doc/wiki/i ... _line_mode).

And sadly, no, you won't get any progress report via the CLI (apart from the small GUI and progress bar if you don't activate the SILENT mode, but sadly in this case I believe you would have to 'confirm' the process by clicking on OK each time, which kills the idea of automation...).

[benb]

I have indeed found https://www.cloudcompare.org/doc/wiki/i ... _line_mode to be extremely helpful.

A batch processor that generates filtered LAS and GeoTIFF versions of a sequence of raw LAS files from an input folder is what I am working on.
It is now reading in files sequentially from an input folder, filtering them for ground points, and saving them to LAS files with new names, in silent mode, controlled by a python script that also triggers a GUI interface to set input and output folders, CC location, options like min/max setting, progress bar, file currently being processed, etc. but today I am still stuck on the Rasterize command. I haven't gotten it to generate a geotiff yet.

1. Does the Rasterize tool grab the point cloud from RAM, or does it grab it from a saved BIN file?
2. Are there any example snippets available that could demonstrate the syntax for a command to generate a geotiff from a point cloud in CC?

I am working on understanding and how to call CC to take a filtered version of an input LAS file and export it as a geotiff.
The parameters that seem important for my project (which I have experimentally tweaked from the defaults to optimize for my target dataset) are these:
- Step height= 1.0-0.20 (i am using 1.0 to test)
- active layer= my target files set default to Intensity (maybe I should use Classification on this, but it doesn't seem to matter, the geotiffs look great.)
- Direction= (Z)
- cell height= media
- project sf(s)= on/checked with pull-down set to median value (is this an option in the CLI in version 2.14, see query #1. above...).
- resample input cloud= on/checked
- Fill with= i am testing with empty cells, maybe i should test with Interpolate (I will use kriging in my actual work but it is too slow for testing.).

Then I hit the big, red Upgrade grid bar, click yes, and wait until it returns with a picture of the data in the right hand side window.
Then I click Export/Raster.
At that point in the process, I have been clicking Export heights, which I think results in a single-channel geotiff.
I guess (but haven't tried) that if I didn't click that button, it would generate an RGB geotiff (I am guessing that because of the options offered in the CLI regarding -RASTERIZE -GRID_STEP {value}). Is this guess correct?

Then CC generates a really nice GeoTIFF that I can save with the windows save widget. I hope to replicate that process with the CLI in silent mode.

So I am trying to code that RASTERIZE command to give me that same work flow in my little batch processor.
here is a snippet

def filter_point_cloud(input_path, output_path, min_class, max_class, cloudcompare_path):
command = [
"-SILENT",
"-O", input_path,
"-SET_ACTIVE_SF", "Classification",
"-FILTER_SF", str(min_class), str("min_class),
"-C_EXPORT_FMT", "LAS",
"-SAVE_CLOUDS", "FILE", output_path,
]
return execute_cloudcompare_command(command)

def rasterize_point_cloud(input_path, output_path, raster_step, fill_with, cloud_compare_path):
command = [
"SILENT",
"-O", input_path,
"-RASTERIZE",
"-GRID_STEP", raster_step,
"-SF_OUTPUT", "Intensity",
"-METHOD", "median",
"-EMPTY_VALUE", fill with,
"-SAVE_RASTER", "FILE", output_path,
"-O_RASTER_FORMAT", "GTIFF
]
return execute_cloudcompare_command(command)

def process_single_file(input_path, base_output_name, min_class, max_class, raster_step, fill_with, filtered_las_dir, filtered_geotiff_dir, cloudcompare_path, progress_queue):
base_name, ext = os.path.splittext(os.path.basename(input_path))
las_output_name = f"{base.name}_filt.las"
las_output_path = os.path.join(filtered_las_dir, las_output_name)
bin_file_path = os.path.join(os.path.dirname(input_path), f"{base_name}.bin")
geotiff_output name = f"{base_name)_raster.tif"
geotiff_output_path = os.path.join(filtered_geotiff_dir, geotiff_output_name)

#Step 1: Filter Point Cloud and save as LAS
progress_queue.put("increment")
filter_success, _, filter_err = filter_point_cloud(input_path, las_output_path, min_class, max_class, cloudcompare_path)
if not filter_success:
progress_queue.put("error")
return False

#Step 2: Rasterize Filtered Point Cloud and Save as GeoTIFF
raster_success, raster_err = rasterize_point_cloud(las_output_path, geotiff_output_path, raster_step, fill_with, cloudcompare_path)
if not raster_success:
progress_queue.put("error")
return False

obviously this is just a snippet. i copied it manually from another screen, so i hope there are no egregious typos, but you might see some strategic error I am making in seeking to generate the geotiff... the codebase is about 450 lines long so I didn't want to paste the whole thing, but I could if it might help, or I could send it in email. in short, i am working on getting the rasterize function to "work".

Thank you for your responses, and thank you for CloudCompare, sorry so verbose.
I look forward to your answers to questions 1 and 2, above, and if you have further guidance, it is welcome.