69 lines
6.5 KiB
Plaintext
69 lines
6.5 KiB
Plaintext
|
o
|
|||
|
|
û\iî+ ã @ s d dl Zd dlZd dlZd dlmZ d dlZd dlZd dl Z d dl
|
|||
|
|
m Z d dl m
|
|||
|
|
Z
|
|||
|
|
mZmZ d dlmZ e eeƒj¡Ze ej¡ e e ej¡¡ de
|
|||
|
|
e e Zh d£Z dZ d Z d
|
|||
|
|
e d e fd d
|
|||
|
|
„Z!de j"j#d e$fdd„Z%ddej#de&d ej#fdd„Z'de j(de d ej#fdd„Z)dS )é N)ÚPath)Ú
|
|||
|
|
output_geobox)Ú SPATIAL_KEYSÚCRS_KEYSÚ
|
|||
|
|
OTHER_KEYS)Úmasking)ÚproductÚtimeÚ
|
|||
|
|
geopolygonÚlikeÚlimitÚensure_locationÚdataset_predicate>/ Zswir_22ZB02Zband_08Zband_8aZB01ZB06Znir_2ÚB12Zband_03Úswir22Zswir_1Únir08Zrededge2Zband_01Zband_04Z
|
|||
|
|
red_edge_3ZB8AZ
|
|||
|
|
red_edge_1Zband_02Zswir_2Znir09ZB07ÚredZnir_3ÚgreenZ
|
|||
|
|
red_edge_2Zband_09Zswir_16ZB09ZB03Zrededge1ÚblueZband_11ÚnirÚB11ÚcoastalZB05ZB08ZB04Zband_05Znir_1Úswir16Zband_07Zrededge3Zcoastal_aerosolZband_06Zband_12g-C ëâ6?gš™™™™™¹¿ÚmatchesÚreturnc C s¤ t d¡}i }| D ]F}|jd }| |¡}|s t d|› �¡ q | d¡› d| d¡› �}t| d¡ƒ}||v rIt|| ƒd |k rH||i||< q ||i||< q |S ) zçFilter for the highest element84 processing sequence number per scene (scene label excluding the sequence number)
|
|||
|
|
|
|||
|
|
: return : { scene_id_excluding_sequence_number : { highest_sequence_number : datacube.model.Dataset }}
|
|||
|
|
z(S2.+)_([0-9]+)_(L2A)Úlabelz/Dataset label does not match expected pattern: é Ú_é é r ) ÚreÚcompileÚ metadata_docÚmatchÚlogÚwarningÚgroupÚintÚlist)r ÚpÚsorterÚdsr ÚmÚkeyÚseq© r/ ú</home/x79/CSIROBoeingPhase5-Vietnam/easi_tools/load_s2l2a.pyÚhighest_sequence_number_ s
|
|||
|
|
|
|||
|
|
|
|||
|
|
€r1 r+ c C sj | j d }| dd¡}t d¡}| |¡}|s!t d|› �¡ dS t| d¡ƒdk r,d S | d
|
|||
|
|
d ¡}| S ) zZReturn True if a dataset's metadata indicates that the offset correction should be appliedÚ
|
|||
|
|
propertieszs2:processing_baselinez0.0z (\d+)\.(\d+)z=Dataset processing_baseline does not match expected pattern: Nr é Fz earthsearch:boa_offset_applied) r" Úgetr r! r# r$ r% r' r& )r+ ÚpropsÚbaseliner) r, Zboa_offset_appliedr/ r/ r0 Úds_requires_offsetx s
|
|||
|
|
|
|||
|
|
|
|||
|
|
r7 Úoffsetc C s^ dd„ | j D ƒ}t | | ¡}|dkr | | |¡t | |< | S | | |¡t | | |< | S )zeApply the scale and offset correction to each reflectance band where there is valid data (not nodata)c S s g | ]}|t v r|‘qS r/ )Ú
|
|||
|
|
refl_bands)Ú.0Úxr/ r/ r0 Ú
|
|||
|
|
<listcomp>� s z,apply_correction_to_data.<locals>.<listcomp>r )Ú data_varsr Úvalid_data_maskÚwhereÚ scale_factor)r+ r8 Z refl_varsÚmaskr/ r/ r0 Úapply_correction_to_data‹ s ÿrB ÚdcÚqueryc C sZ | dd¡}|dkrt d|› �¡ dS d}d|v r!|d }|d= |du r6dd„ | ¡ D ƒ}| jdi |¤Ž}d |vr>d
|
|||
|
|
|d < t|ƒ}g g }}| ¡ D ]"}t|| ¡ ƒd } t | ƒ}
|
|||
|
|
|
|
|||
|
|
du r`qK|
|
|||
|
|
rh|
|
|||
|
|
| ¡ qK|
|
|||
|
|
| ¡ qK|| } d} t |ƒd krƒt d ¡ d
|
|||
|
|
}
|
|||
|
|
t
|
|||
|
|
} t |ƒd kr’t d¡ d}
|
|||
|
|
d } | dur¬| jdd| i|¤Ž}t|| ƒ}t |
|
|||
|
|
¡ |S t d¡ t d¡ tdd| i|¤Ž}d}d|v rÚ|d }| dd¡dkrÚ|d ddi¡ dD ] }||v rå||= qÜ| jd||dœ|¤Ž}| jd||dœ|¤Ž}t|ƒ}t|t
|
|||
|
|
ƒ}tj||gdd�}| d¡}|du�r&| dd¡dk�r&| |¡}t d
|
|||
|
|
¡ |S )a_
|
|||
|
|
Replaces datacube.load(**query) for s2_l2a products.
|
|||
|
|
|
|||
|
|
Method:
|
|||
|
|
- Find all datasets matching the query (dc.find_datasets)
|
|||
|
|
- Filter for the highest element84 processing sequence number per scene (scene label excluding the sequence number)
|
|||
|
|
- Filter into two lists for datasets that have
|
|||
|
|
- "s2:processing_baseline" >= "04.00" and "earthsearch:boa_offset_applied" == False (offset correction required)
|
|||
|
|
- everything else (no correction required)
|
|||
|
|
- If either list is empty then load the non-empty list, apply scale (and offset if required), and return the xarray Dataset
|
|||
|
|
- Load and combine the two lists of datasets
|
|||
|
|
- Load each list, apply scale (and offset if required)
|
|||
|
|
- Concat on time dimension and sort by time
|
|||
|
|
- Return the combined xarray Dataset
|
|||
|
|
|
|||
|
|
Notes:
|
|||
|
|
- Any 'groupby' function is applied to each of the xarray Datasets prior to them being combined.
|
|||
|
|
This could create "extra" (non-grouped) time layers in the combined Dataset if the groupby function
|
|||
|
|
would have grouped datasets (scenes) from both lists.
|
|||
|
|
- Scale and offset are applied to the reflectance bands where there is valid data (not `nodata`).
|
|||
|
|
This includes applying the "scale_factor" even if no datasets require the offset correction.
|
|||
|
|
Other masks can be applied by the user (e.g. pixel quality or cloud masking).
|
|||
|
|
r z<all products>Ús2_l2az9This function only applies to the "s2_l2a" product, not: NÚdatasetsc S s i | ] \}}|t v r||“qS r/ )Ú search_keys)r: ÚkÚvr/ r/ r0 Ú
|
|||
|
|
<dictcomp>¾ s z*load_s2l2a_with_offset.<locals>.<dictcomp>Úskip_broken_datasetsTr z&All datasets require offset correctionzPThe valid_data_mask, scale and offset have been applied to the reflectance bandsz%No datasets require offset correctionzTThe valid_data_mask and scale (no offset) have been applied to the reflectance bandsz9Mix of datasets found with either offset required or not.zQWe will load two xarrays, apply offset where required, and merge into one xarray.Ú dask_chunksr r )Ú
|
|||
|
|
output_crsÚ
|
|||
|
|
resolutionÚalign)rF r )Údimr/ )r4 r$ ÚerrorÚitemsÚ
|
|||
|
|
find_datasetsr1 Úkeysr( Úvaluesr7 ÚappendÚlenÚinfoÚ
|
|||
|
|
add_offsetÚloadrB r ÚupdateÚxrÚconcatÚsortbyÚchunk)rC rD r r Z
|
|||
|
|
search_paramsr* Zoffset_appliedZoffset_requiredr- r+ Z
|
|||
|
|
isrequiredZmatches_combinedZ this_offsetÚmsgÚdataÚxxZ
|