21
votes
QGIS slow performance with PostgreSQL/PostGIS
This is a broad question... here are a few things I have gathered through the years:
Make sure the geom/geog column is using a single type (ex avoid geometry but instead use geometry(point,4326)). ...
- 44.4k
19
votes
Accepted
Explaining long computing time of Python code
Here are some issues with your script:
I've already learned that using GeoJSON for analyzing is a too bad idea. The script below that takes 20 seconds when I use GeoPackage takes about 8-10 hours ...
- 78.6k
18
votes
Finding nearest neighbor for each point in the same table
This is essentially a duplicate question of multiple others, with the sole difference being a table self-join.
However, all queries currently present in this post have delicate CRS misunderstandings, ...
- 31.5k
15
votes
Faster calculation of cumulative sum in attribute table in QGIS
You can use the script below in QGIS Python Editor. It takes about 45 seconds (including saving to the source file) for 1.5 million features in SSD or HDD. Select the layer first.
import time
from ...
- 78.6k
14
votes
Accepted
QGIS form performance
It seems to be a bug in QGIS that has existed for some time. I've tested versions 2.18, 3.4 and 3.10.
When using a bigint as a primary key, QGIS will cast the primary key to a text field. This causes ...
- 1,387
14
votes
Accepted
Speed up reading GeoPackage as GeoDataFrame with Python
Install and use the optional pyogrio I/O engine to read the data... that will be a lot faster. Adding the use_arrow=True parameter as well will give another big performance improvement, but then you'...
- 4,657
12
votes
Fastest way to convert big raster to polyline using R or Python?
Edit: In the latest version of sf (v0.9.1) and stars (v0.4.1), you need to specify the merge = TRUE argument in the call to sf::st_as_stars() function.
For posterity, I've been having success with ...
- 315
12
votes
Accepted
What can be a reason of QGIS slow work?
First, I would try with a fresh profile. Go to Settings -> User Profiles - New Profile. This is basically like a fresh QGIS install - no Plugins are loaded, no (maybe wrong) settings you changed ...
- 3,169
11
votes
Accepted
Only union/dissolve intersecting or adjacent features to speed up query
I would look into using ST_ClusterDBSCAN. I have had tremendous success using this function to solve many cluster like geometric problems.
WITH clusters
AS(select st_clusterdbscan(geom, 0, 2) over()...
- 4,576
11
votes
Disjoint is very slow in Virtual Layer but fast on symbology
The issue is the cross join on non-intersection. Your query reads:
for every row in g, return every row in s that does not intersect with the current g
which seems in fact what you want, but about ...
- 31.5k
10
votes
Accepted
Measuring loading time of each layer when opening a QGIS project
Since QGIS 3.16 you can know the time it takes to load each layer from a QGIS project.
Go to View --> Panels --> Debugging/Development Tools and click on the Profiler in the left menu.
As soon ...
- 37.3k
9
votes
ArcPy UpdateCursor auto pausing after every 1000 records
By default the auto commit interval is set to 1000. You can change the interval using the following:
arcpy.env.autoCommit = 5000
- 116
8
votes
Increasing speed of crop, mask, & extract raster by many polygons in R?
This question is very old so my answer is just for posterities. Since 2020-11-20 there is the new exactextractr package which implement the exact_extract() function, which is the new benchmark for ...
- 297
8
votes
Accepted
Importing CSV (via VRT) to PostGIS is slow with -skipfailures?
You don't have to use a .vrt any more. ogr2ogr supports reading csv files with geometry directly since version 2.1.
The ogr2ogr command:
ogr2ogr -f "PostgreSQL" PG:"host=000.000.000.000 port=0000 ...
- 17.1k
8
votes
Accepted
Fastest way to extract a raster in R (improve the time of my reproducible code)
I get much faster results with velox if I crop the raster before running extract, e.g.:
r <- velox("testras_so.tif")
r$crop(poly)
r$extract(poly)
I've also been working on a package with ...
- 13.3k
8
votes
Accepted
Choosing raster format for QField
The QField documentation recommends using COG raster for this purpose.
Other sources (citation needed) recommend mbtiles format.
Whatever you choose, here are a couple of things to consider:
Tiling ...
- 28.4k
8
votes
Accepted
Accelerating GeoPandas for selecting points inside polygon
With the help of @Taras I could solve my efficiency problem.
As the first try, I thought maybe the tqdm library is causing this delay. So I modified my code to
within_points = [points.geometry[i]....
- 251
8
votes
Multi-threaded compression in rasterio?
Yes, although this seems not to be mentioned in the documentation.
If you include num_threads=8 or num_threads='all_cpus' as an argument to rasterio.open then multithreading will be enabled (for ...
- 496
8
votes
Accepted
Left joining CSV to GeoPackage file and outputting GeoJSON in scalable way with ogr2ogr
One possible solution is to load the CSV file to the GPKG file as a new layer. Once it's in the GPKG, there is no need for INDIRECT_SQLITE! So:
ogr2ogr -update base.gpkg join.csv -nln csv
ogr2ogr ...
- 3,276
8
votes
Accepted
Efficiently check if polygon contains any point from a list
This is a typical case to use a spatial index.
Shapely has support for spatial indexes via shapely.STRtree.query
I did a quick test with 1000 polygons and 750 points derived of them... and even with ...
- 4,657
7
votes
Prevent SAGA in QGIS 3.0 from creating certain outputs?
Should be possible to do if you feel like hacking your qgis install.
look for this file in your qgis install: https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/saga/description/...
- 28.1k
7
votes
QGIS more than 3000 raster works slowly
You may see considerable benefit if you load them into a single virtual raster (vrt). You can do that through the processing toolbox, by searching for "build vrt"
- 28.1k
7
votes
Accepted
PostGIS ST_Intersects is slow and doesn't seem to use spatial index
Try
WITH
poly AS (
SELECT geom
FROM admin_boundary
WHERE id = <id>
)
UPDATE highway_only_motor AS a
SET has_business_value = true
FROM poly AS b
WHERE ...
- 31.5k
7
votes
Shapely with Rtree versus STRtree
The poor performance was the result of a bug in Shapely. A fix has been committed and will be available in the next version.
Please find the speed comparison from shapely version 2.0.1 below:
Seconds ...
- 13.3k
7
votes
Accepted
Complex iteration in QGIS geometry generator for compass rose with wind speed data
To do so, proceed as follows: create a variable that stores an array of all fieldnames you want to use. You can define it as a project variable (probably more efficient, as it is not re-created for ...
- 80.2k
6
votes
Accepted
Displaying huge geotiffs (or vrts) with QGIS?
Ok, well I solved both my problems...mainly by buying an NVMe SSD. My disk read/write has gone from 125 MB/s to 1200 MB/s.
Programatically, there are a few things you can do to help your read/write ...
- 2,954
6
votes
Displaying huge geotiffs (or vrts) with QGIS?
You seem to have two main concerns: VRT id slow with browsing and it is slow to build global overviews.
While I am sure that GDAL VRT used to be slow for me and my MapServer many years ago it may be ...
- 71.7k
6
votes
Improving poor performance of ArcPy update cursor with ArcSDE and Editor?
The nested cursors are slowing you way down. Instead, use a searh cursor to create a dictionary with UIDs as your key and rows as your values. Then use an update cursor to update your second feature ...
- 13.9k
6
votes
Accepted
Which ArcGIS Online basemap has the best performance in ArcMap?
tl;dr; The performance of all basemaps from arcgis.com inside ArcMap will pretty much be equal.
The basemaps you reference all come from the same cloud infrastructure and show the same thing: static ...
- 17.1k
6
votes
Accepted
Slow loading of QGIS Project
Did you try out the Debugging/Development Tools? They are availably since QGIS 3.16.
The tool analyses the time it takes to load each layer from a QGIS project.
Go to View --> Panels --> ...
- 2,440
Only top scored, non community-wiki answers of a minimum length are eligible