Getting Started¶
Installation and Setup¶
Installing the package should be super duper simple as we utilize Python’s setuptools.
$ pipenv install bethesda-structs
$ # or if you're old school...
$ pip install bethesda-structs
Or you can build and install the package from the git repo.
$ git clone https://github.com/stephen-bunn/bethesda-structs.git
$ cd ./bethesda-structs
$ python setup.py install
Example Usage¶
BaseFiletype
.BaseArchive
and BasePlugin
and any other one-off file types.Because this package is just a parser around some of Bethesda’s popular file formats, its hopefully extendable to different purposes you might have. Below are a few quick examples of how you might use this package, but remember to keep the autodocs handy!
Checking if a file can be parsed¶
Because many of the structures used to parse these filetypes are bulit using construct, the parser silently fails if parsing raises an error.
For this reason, all file types include the classmethod can_handle()
, which takes a filepath and returns either True
or False
depending on if the filetype class believes it can parse the file.
This method can be executed similar to the following:
>>> from bethesda_structs.archive import BSAArchive
>>> BSAArchive.can_handle('/media/sf_VMShared/bsa/Campfire.bsa')
True
>>> from bethesda_structs.plugin import FNVPlugin
>>> FNVPlugin.can_handle('/media/sf_VMShared/bsa/Campfire.bsa')
False
Parsing a file¶
We try to keep the api simple and consistent across all of our parsable filetypes. For example, the following methods should be available on all parseable filetypes:
parse()
— parses bytesparse_stream()
— parses bytes from a file streamparse_file()
— parses bytes from a file path
>>> from bethesda_structs.archive import BSAArchive
>>> archive = BSAArchive.parse_file('/media/sf_VMShared/bsa/Campfire.bsa')
>>> archive
BSAArchive(filepath=PosixPath('/media/sf_VMShared/bsa/Campfire.bsa'))
>>> archive.filepath
PosixPath('/media/sf_VMShared/bsa/Campfire.bsa')
The filepath
attribute is automatically transformed into a pathlib.Path
instance on initialization of the filetype.
This is really helpful for quickly obtaining file information if you need it.
>>> archive.filepath.stat()
os.stat_result(st_mode=33272, st_ino=6, st_dev=46, st_nlink=1, st_uid=0, st_gid=999, st_size=25097317, st_atime=1522860401, st_mtime=1522475016, st_ctime=1522475016)
Note
Parsed files only have the filepath
attribute populated if either the parse_file()
method was used, or the filepath
named argument was passed into the other parsing methods.
Examining a parsed file¶
All parsed files should contain the container
attribute which is a root Container
instance.
This contains the required information for examining a parsed file’s contents.
For example you can view the header sub-container of a BSAArchive
like the following:
>>> from bethesda_structs.archive import BSAArchive
>>> archive = BSAArchive.parse_file('/media/sf_VMShared/bsa/Campfire.bsa')
>>> print(archive.container.header)
Container:
magic = b'BSA\x00' (total 4)
version = 105
directory_offset = 36
archive_flags = Container:
directories_named = True
files_named = True
directory_count = 4
file_count = 493
directory_names_length = 50
file_names_length = 14839
file_flags = Container:
BaseFiletype
most likely implements a different container
structure.BSAArchive
has directory_records
, directory_blocks
, and file_names
sub-containers:>>> from bethesda_structs.archive import BSAArchive
>>> archive = BSAArchive.parse_file('/media/sf_VMShared/bsa/Campfire.bsa')
>>> print(archive.container.directory_records)
ListContainer:
Container:
hash = 1948419268744733541
file_count = 155
name_offset = 14971
Container:
hash = 2736503539341685349
file_count = 174
name_offset = 17467
Container:
hash = 3940292978845119603
file_count = 163
name_offset = 20268
Container:
hash = 11606842737777340531
file_count = 1
name_offset = 22885
>>> print(archive.container.directory_blocks[3]) # contains 1 file
Container:
name = u'meshes\\mps\x00' (total 11)
file_records = ListContainer:
Container:
hash = 16183754957220078963l
size = 2384
offset = 25094933
>>> print(archive.container.file_names[:5])
['_camp_objectplacementindicatorthread01.psc', '_camp_objectplacementindicatorthread02.psc', '_camp_objectplacementindicatorthread03.psc', '_camp_tentsitlayscript.psc', 'campcampfire.psc']
However, BTDXArchive
has a completely different structure just using a files
attribute.
>>> from bethesda_structs.archive import BTDXArchive
>>> archive = BTDXArchive.parse_file('/media/sf_VMShared/ba2/Immersive HUD - Main.bsa')
>>> print(archive.container.header)
Container:
magic = b'BTDX' (total 4)
version = 1
type = u'GNRL' (total 4)
file_count = 16
names_offset = 39979
>>> print(archive.container.files[0])
Container:
hash = 2246534376
ext = u'swf' (total 3)
directory_hash = 3539859571
offset = 600
packed_size = 0
unpacked_size = 5011
So knowledge of what you are parsing is important, but how it’s being done shouldn’t be.
Extracting an archive¶
All subclasses of BaseArchive
contain a method extract()
.
This method makes it incredibly simple to extract the content of a parsed archive into a directory.
>>> from bethesda_structs.archive import BSAArchive
>>> archive = BSAArchive.parse_file('/media/sf_VMShared/bsa/Campfire.bsa')
>>> archive.extract('/home/stephen-bunn/Downloads/campfire-extracted/')
>>> from pathlib import Path
>>> for path in pathlib.Path('/home/stephen-bunn/Downloads/campfire-extracted/').glob('**/*'):
... print(path.as_posix())
/home/stephen-bunn/Downloads/campfire-extracted/scripts/_camp_indicatortrigger.pexl
/home/stephen-bunn/Downloads/campfire-extracted/scripts/_camp_objectplacementthread30.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/campconjuredshelter.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/_camp_objectplacementindicatorthread02.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/_camp_instinctseffects.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/_camp_legacymenu.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/_camp_objectplacementindicatorthread.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/bladessparringscript.pex
/home/stephen-bunn/Downloads/campfire-extracted/scripts/tentsystem.pex
... <only first 9 files> ...
Both variants of BTDXArchive
can also be extracted (GNRL
and DX10
).
Getting extraction progress¶
The extract()
method takes a named argument progress_hook
that acts as a (pre/post) callback function taking 3 positional arguments:
current
— current extracted bytestotal
— total bytes to extractfilepath
— current filepath being extracted
>>> from bethesda_structs.archive import BSAArchive
>>> archive = BSAArchive.parse_file('/media/sf_VMShared/bsa/Campfire.bsa')
>>> def progress_hook(current, total, filepath):
... print(((current / total) * 100.0, filepath))
>>> archive.extract('/home/stephen-bunn/Downloads/campfire-extracted/', progress_hook=progress_hook)
(0.0, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_objectplacementindicatorthread01.psc')
(0.0003748842843881753, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_objectplacementindicatorthread01.psc')
(0.0003748842843881753, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_objectplacementindicatorthread02.psc')
(0.0007497685687763506, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_objectplacementindicatorthread02.psc')
(0.0007497685687763506, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_objectplacementindicatorthread03.psc')
(0.0011246528531645259, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_objectplacementindicatorthread03.psc')
(0.0011246528531645259, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_tentsitlayscript.psc')
(0.004051940775940278, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/_camp_tentsitlayscript.psc')
(0.004051940775940278, '/home/stephen-bunn/Downloads/campfire-extracted/scripts/source/campcampfire.psc')
... <only first 9 callbacks> ...
Working with plugins¶
So far in these quick examples we’ve only seen examples using archives, but we can also parse and examine plugins as well! Well parsing these file types is done in the same way as archives.
>>> from bethesda_structs.plugin import FNVPlugin
>>> plugin = FNVPlugin.parse_file('/media/sf_VMShared/esp/fnv/NVWillow.esp')
>>> print(plugin.container.header)
Container:
type = u'TES4' (total 4)
data_size = 163
flags = Container:
master = True
id = 0
revision = 0
version = 15
data = b'HEDR\x0c\x00\x1f\x85\xab?\x97\x12\x00\x00#\xad'... (truncated, total 163)
subrecords = ListContainer:
Container:
type = u'HEDR' (total 4)
data_size = 12
data = b'\x1f\x85\xab?\x97\x12\x00\x00#\xad\r\x00' (total 12)
parsed = Container:
value = Container:
version = 1.340000033378601
num_records = 4759
next_object_id = 896291
description = u'Header' (total 6)
Container:
type = u'CNAM' (total 4)
data_size = 9
data = b'llamaRCA\x00' (total 9)
parsed = Container:
value = u'llamaRCA' (total 8)
description = u'Author' (total 6)
Container:
type = u'SNAM' (total 4)
data_size = 16
data = b'NVWillow v.1.10\x00' (total 16)
parsed = Container:
value = u'NVWillow v.1.10' (total 15)
description = u'Description' (total 11)
Container:
type = u'MAST' (total 4)
data_size = 14
data = b'FalloutNV.esm\x00' (total 14)
parsed = Container:
value = u'FalloutNV.esm' (total 13)
description = u'Master Plugin' (total 13)
Container:
type = u'DATA' (total 4)
data_size = 8
data = b'\x00\x00\x00\x00\x00\x00\x00\x00' (total 8)
parsed = Container:
value = 0
description = u'File Size' (total 9)
Container:
type = u'ONAM' (total 4)
data_size = 68
data = b'V\xe3\x0c\x00\xc3\xe3\x0c\x00\xc4\xe3\x0c\x00\xc5\xe3\x0c\x00'... (truncated, total 68)
parsed = Container:
value = ListContainer:
844630
844739
844740
844741
1372461
1372463
1383111
1385321
1387301
1387302
1387303
1387304
1387906
1457771
1479505
1520201
1544392
description = u'Overridden Records' (total 18)
Getting plugin masters¶
One of the most common tasks in plugin analysis is determining the masters of a plugin.
The names of a plugin’s masters are stored within the MAST
subrecords in the plugin’s header record.
Using the information above, you can get these names like the following:
>>> from bethesda_structs.plugin import FNVPlugin
>>> masters = []
>>> for subrecord in plugin.container.header.subrecords:
... if subrecord.type == 'MAST':
... masters.append(subrecord.parsed.value)
>>> print(masters)
['FalloutNV.esm']
You can also use the iter_subrecords()
helper method to simplify your code:
>>> masters = [
... subrecord.parsed.value
... for subrecord in plugin.iter_subrecords(
... 'MAST', 'TES4',
... include_header=True
... )
... ]
>>> print(masters)
['FalloutNV.esm']
Getting key (KEYM
) records¶
Here is a quick example at getting the data for the first KEYM
record (an in-game key).
This probably really isn’t that helpful to you, but I think an example was needed on how to iterate over specific records (as they can become quite large).
>>> from bethesda_structs.plugin import FNVPlugin
>>> for record in plugin.iter_records('KEYM'):
... print(record)
... break
Container:
type = u'KEYM' (total 4)
data_size = 279
flags = Container:
id = 17415634
revision = 0
version = 15
data = b'EDID\x17\x00WillowNova'... (truncated, total 279)
subrecords = ListContainer:
Container:
type = u'EDID' (total 4)
data_size = 23
data = b'WillowNovacBunga'... (truncated, total 23)
parsed = Container:
value = u'WillowNovacBungalowKey' (total 22)
description = u'Editor ID' (total 9)
Container:
type = u'OBND' (total 4)
data_size = 12
data = b'\xff\xff\xfc\xff\x00\x00\x01\x00\x04\x00\x00\x00' (total 12)
parsed = Container:
value = Container:
X1 = -1
Y1 = -4
Z1 = 0
X2 = 1
Y2 = 4
Z2 = 0
description = u'Object Bounds' (total 13)
Container:
type = u'FULL' (total 4)
data_size = 27
data = b'Dino Dee-lite Bu'... (truncated, total 27)
parsed = Container:
value = u'Dino Dee-lite Bungalow Key' (total 26)
description = u'Name' (total 4)
Container:
type = u'MODL' (total 4)
data_size = 23
data = b'Clutter\\Key01Dir'... (truncated, total 23)
parsed = Container:
value = u'Clutter\\Key01Dirty.NIF' (total 22)
description = u'Model Filename' (total 14)
Container:
type = u'ICON' (total 4)
data_size = 48
data = b'Interface\\Icons\\'... (truncated, total 48)
parsed = Container:
value = u'Interface\\Icons\\PipboyImages\\Ite'... (truncated, total 47)
description = u'Large Icon Filename' (total 19)
Container:
type = u'MICO' (total 4)
data_size = 66
data = b'Interface\\Icons\\'... (truncated, total 66)
parsed = Container:
value = u'Interface\\Icons\\PipboyImages_sma'... (truncated, total 65)
description = u'Small Icon Filename' (total 19)
Container:
type = u'SCRI' (total 4)
data_size = 4
data = b'T.\n\x01' (total 4)
parsed = Container:
value = FormID(form_id=17444436, forms=['SCPT'])
description = u'Script' (total 6)
Container:
type = u'YNAM' (total 4)
data_size = 4
data = b'\xbb\x10\x07\x00' (total 4)
parsed = Container:
value = FormID(form_id=463035, forms=['SOUN'])
description = u'Sound - Pick Up' (total 15)
Container:
type = u'ZNAM' (total 4)
data_size = 4
data = b'\xbc\x10\x07\x00' (total 4)
parsed = Container:
value = FormID(form_id=463036, forms=['SOUN'])
description = u'Sound - Drop' (total 12)
Container:
type = u'DATA' (total 4)
data_size = 8
data = b'\x00\x00\x00\x00\x00\x00\x00\x00' (total 8)
parsed = Container:
value = Container:
value = 0
weight = 0.0
description = u'Data' (total 4)