Commit Graph

1162 Commits

Author SHA1 Message Date
ff3887757e Add internal tests for socket apis (#1900) 2023-03-09 12:31:06 +08:00
128c0ea899 Add internal tests for WASI threads (#1963)
Add internal tests for WASI threads. These tests are run in addition to
the ones in the proposal:
https://github.com/WebAssembly/wasi-threads/tree/main/test/testsuite.

The purpose is to test additional and more complex scenarios.
2023-03-09 09:03:16 +08:00
289fc5efbf Enhance/Fix sample socket-api and workload (#2006)
- Add python script to test socket-api cases
- Fix issues in socket-api send_recv wasm app
- Fix issues in building samples/workload/meshoptimizer
- Enhance build script of sample workload
2023-03-08 16:36:08 +08:00
a15a731e12 wasi-nn: Support multiple TFLite models (#2002)
Remove restrictions:
- Only 1 WASM app at a time
- Only 1 model at a time
   - `graph` and `graph-execution-context` are ignored

Refer to previous document:
e8d718096d/core/iwasm/libraries/wasi-nn/README.md
2023-03-08 15:54:06 +08:00
f279ba84ee Fix multi-threading issues (#2013)
- Implement atomic.fence to ensure a proper memory synchronization order
- Destroy exec_env_singleton first in wasm/aot deinstantiation
- Change terminate other threads to wait for other threads in
  wasm_exec_env_destroy
- Fix detach thread in thread_manager_start_routine
- Fix duplicated lock cluster->lock in wasm_cluster_cancel_thread
- Add lib-pthread and lib-wasi-threads compilation to Windows CI
2023-03-08 10:57:22 +08:00
7a3d2bfab6 Fix os_socket_addr_resolve() for IPv6 (#2015)
The function always specified IPv4 socklen to sockaddr_to_bh_sockaddr(),
therefore the assertion was failing; however, sockaddr_to_bh_sockaddr()
never actually used socklen parameter, so we deleted it completely.
2023-03-08 07:08:12 +08:00
6a3ae689f7 nuttx: Add missing pthread.h header (#2009) 2023-03-07 22:01:16 +08:00
76b8225dac Fix key error in build_llvm.py (#2014) 2023-03-07 17:48:26 +08:00
e8fe2d1c42 support SSH for git clone llvm (#2011)
Co-authored-by: Kevin Kang <wilson.qch1685@gmail.com>
2023-03-07 17:05:36 +08:00
2a5a2243e3 ci: Setup wasi-sdk for compilation on nuttx to fix break (#2012) 2023-03-07 12:06:38 +08:00
5a7fbda7ac Enable to collect code coverage of samples/simple (#2003) 2023-03-07 08:45:03 +08:00
04616d398d Fix wasm_cluster_create_thread issue (#2004)
In wasm_cluster_create_thread, the new_exec_env is added into the cluster's
exec_env list before the thread is created, so other threads can access the
fields of new_exec_env once the cluster->lock is unlocked, while the
new_exec_env's handle is set later inside the thread routine. This may result
in the new_exec_env's handle be invalidly accessed by other threads.
2023-03-06 18:51:13 +08:00
e6a0184797 enable custom llvm build flags (#2000) 2023-03-06 06:50:53 +08:00
e8d718096d Add/reorganize locks for thread synchronization (#1995)
Attempt to fix data races when using threads.
- Protect access (from multiple threads) to exception and memory
- Fix shared memory lock usage
2023-03-04 08:15:26 +08:00
1c44411a97 Add timeout to CI WASI tests and update WASI testsuite commit (#1997)
- Add timeout to CI WASI tests to avoid keeping CI node busy in case of
  deadlock in thread tests
- Update wasi-testsuite commit used, after fix in wasi-threads proposal
  https://github.com/WebAssembly/wasi-threads/pull/40
2023-03-03 16:30:59 +08:00
1be202fad8 Fix several issues found (#1996)
- CMakeLists.txt: add lib_export.h to install list
- Fast JIT: enlarge spill cache size to enable several standalone cases
                when hw bound check is disabled
- Thread manager: wasm_cluster_exit_thread may destroy an invalid
               exec_env->module_inst when exec_env was destroyed before
- samples/socket-api: fix failure to run timeout_client.wasm
- enhance CI build wasi-libc and sample/wasm-c-api-imports CMakeLlist.txt
2023-03-03 15:00:54 +08:00
9f0c4b63ac doc/memory_usage.md: Update after dev/wasi_threads merge (#1994) 2023-03-01 05:19:34 +08:00
9b9ae0cfac Update cmake files and wamr-test-suites to support collect code coverage (#1992)
Support collecting code coverage with wamr-test-suites script by using
lcov and genhtml tools, eg.:
  cd tests/wamr-test-suites
  ./test_wamr.sh -s spec -b -P -C

The default code coverage and html files are generated at:
  tests/wamr-test-suites/workspace/wamr.lcov
  tests/wamr-test-suites/workspace/wamr-lcov.zip

And update wamr-test-suites scripts to support testing GC spec cases to
avoid frequent synchronization conflicts between branch main and dev/gc.
2023-02-28 17:38:18 +08:00
b4f0228497 Python WAMR API binding: Add malloc/free and register_native (#1989) 2023-02-28 16:19:17 +08:00
4ca57a0228 Update WASI thread proposal tests in CI (#1985)
Update CI to run extended WASI threads tests included in the proposal
https://github.com/WebAssembly/wasi-testsuite/tree/prod/testsuite-all/tests/proposals/wasi-threads
2023-02-27 20:28:37 +08:00
52e26e59cf Add lock to protect the operations of accessing exec env (#1991)
Data race may occur when accessing exec_env's fields, e.g. suspend_flags
and handle. Add lock `exec_env->wait_lock` for them to resolve the issue.
2023-02-27 19:53:41 +08:00
38c67b3f48 thread-mgr: Fix spread "wasi proc exit" exception and atomic.wait issues (#1988)
Raising "wasi proc exit" exception, spreading it to other threads and then
clearing it in all threads may result in unexpected behavior: the sub thread
may end first, handle the "wasi proc exit" exception and clear exceptions
of other threads, including the main thread. And when main thread's
exception is cleared, it may continue to run and throw "unreachable"
exception. This also leads to some assertion failed.

Ignore exception spreading for "wasi proc exit" and don't clear exception
of other threads to resolve the issue.

And add suspend flag check after atomic wait since the atomic wait may
be notified by other thread when exception occurs.
2023-02-24 20:05:39 +08:00
63273a1673 Fix issues in the libc-wasi poll_oneoff when thread-mgr is enabled (#1980)
Fix issues in the libc-wasi `poll_oneoff` when thread manager is enabled:
-  The exception of a thread may be cleared when other thread runs into
   `proc_exit` and then calls `clear_wasi_proc_exit_exception`, so should not
   use `wasm_runtime_get_exception` to check whether an exception was
    thrown, use `wasm_cluster_is_thread_terminated` instead
- We divided one time poll_oneoff into many times poll_oneoff to check
   the exception to avoid long time waiting in previous PR, but if all events
   returned by one time poll are all waiting events, we need to continue to
   wait but not return directly.

Follow-up on #1951. Tested with multiple timeout values, with and without
interruption and measured the time spent sleeping.
2023-02-23 16:11:33 +08:00
92c4bbebad Fix getting port issue in posix os_socket_bind (#1981)
In the previous code, the `*port` is assigned before `getsockname`, so the caller
may be not able to get the actual port number assigned by system.
Move the assigning of `*port` to be after `getsockname` to resolve the issue.
2023-02-22 18:59:13 +08:00
d2772c4153 Re-org calling post instantiation functions (#1972)
- Use execute_post_instantiate_functions to call start, _initialize,
  __post_instantiate, __wasm_call_ctors functions after instantiation
- Always call start function for both main instance and sub instance
- Only call _initialize and __post_instantiate for main instance
- Only call ___wasm_call_ctors for main instance and when bulk memory
  is enabled and wasi import functions are not found
- When hw bound check is enabled, use the existing exec_env_tls
  to call func for sub instance, and switch exec_env_tls's module inst
  to current module inst to avoid checking failure and using the wrong
  module inst
2023-02-22 12:24:11 +08:00
bc60064de5 Fix data races in atomic wait/notify and interp goto table (#1971)
Add shared memory lock when accessing the address to atomic wait/notify
inside linear memory to resolve its data race issue.

And statically initialize the goto table of interpreter labels to resolve the
data race issue of accessing the table.
2023-02-22 11:43:49 +08:00
0fa0813a5a Fix issues reported by Coverity static analysis (#1974) 2023-02-22 11:10:21 +08:00
bb5629811f language-bindings/python/wamr-api: Fix a sample (#1977) 2023-02-21 18:36:03 +08:00
d8c333f32c Fix wamrapi python binding for darwin (#1978) 2023-02-21 18:35:15 +08:00
91eafa1ead Make a workaround for EGO when fstat returns NOT_SUPPORT (#1970)
The problem was found by a `Golang + WAMR (as CGO)` wrapped by EGO
in SGX Enclave.

`fstat()` in EGO returns dummy values:
- EGO uses a `mount` configuration to define the mount points that apply
  the host file system presented to the Encalve.
- EGO has a different programming model: the entire application runs inside
  the enclave. Manual ECALLs/OCALLs by application code are neither
  required nor possible.

Add platform ego and add macro control for the return value checking of
`fd_determine_type_rights` in libc-wasi to resolve the issue.
2023-02-21 11:11:27 +08:00
37b09d0f24 Expose wasm_runtime_call_indirect (#1969)
The function has been there for long. While what it does look a bit unsafe
as it calls a function which may be not wasm-wise exported explicitly, it's
useful and widely used when implementing callback-taking APIs, including
our pthread_create's implementation.
2023-02-20 18:56:55 +08:00
e516de8ec7 Fix possible data race in thread manager (#1973)
Destroy child thread's exec_env before destroying its module instance and
add the process into cluster's lock to avoid possible data race: if exec_env
is removed from custer's exec_env_list and destroyed later, the main thread
may not wait it and start to destroy the wasm runtime, and the destroying
of the sub thread's exec_env may free or overread/written an destroyed or
re-initialized resource.

And fix an issue in wasm_cluster_cancel_thread.
2023-02-20 17:11:21 +08:00
739acfc908 Merge pull request #1966 from bytecodealliance/dev/wasi_threads
Merge wasi_threads to main
2023-02-17 17:42:47 +08:00
ef3a683392 Don't call start/initialize in child thread's instantiation (#1967)
The start/initialize functions of wasi module are to do some initialization work
during instantiation, which should be only called one time in the instantiation
of main instance. For example, they may initialize the data in linear memory,
if the data is changed later by the main instance, and re-initialized again by
the child instance, unexpected behaviors may occur.

And clear a shadow warning in classic interpreter.
2023-02-17 15:11:05 +08:00
e170c355a2 Merge branch main into dev/wasi_threads 2023-02-17 08:46:12 +08:00
50650e4634 Modify poll_oneoff in libc-wasi to make it interruptible (#1951) 2023-02-16 21:03:33 +08:00
3cc132e8fc Add WAMR API bindings in Python (#1959)
Before adding the new bindings:
1. Moved wasm-c-api in a subfolder wasmcapi in the package.
2. Adapted the tests to be able to run in this new structure.

New:
1. Added the WAMR API in another folder wamrapi in the same level as wasm-c-api.
2. Created an OOP proposal.
3. Added an example using this proposal.
2023-02-16 15:21:28 +08:00
f60c3c6111 Add Multi-tier JIT tests in Ubuntu CI (#1964) 2023-02-16 14:16:41 +08:00
216dc43ab4 Use shared memory lock for threads generated from same module (#1960)
Multiple threads generated from the same module should use the same
lock to protect the atomic operations.

Before this PR, each thread used a different lock to protect atomic
operations (e.g. atomic add), making the lock ineffective.

Fix #1958.
2023-02-16 11:54:19 +08:00
1c17665f68 Fix libc-wasi build failure when using clang (#1950)
Resolves #1949
2023-02-14 18:28:00 +08:00
fd487fbe99 CI: Enable Multi-tier JIT by default for released iwasm binary (#1957) 2023-02-14 11:19:52 +08:00
531771eb11 Don't terminate other threads when create thread failed (#1955) 2023-02-13 16:23:24 +08:00
3698f2279b Improve wasm-c-api instantiation-time linking (#1902)
Add APIs to help prepare the imports for the wasm-c-api `wasm_instance_new`:
- wasm_importtype_is_linked
- wasm_runtime_is_import_func_linked
- wasm_runtime_is_import_global_linked
- wasm_extern_new_empty

For wasm-c-api, developer may use `wasm_module_imports` to get the import
types info, check whether an import func/global is linked with the above API,
and ignore the linking of an import func/global with `wasm_extern_new_empty`.

Sample `wasm-c-api-import` is added and document is updated.
2023-02-13 15:06:04 +08:00
427abf02c8 Fix potential integer overflow issue in wasm-c-api (#1954)
Fix potential integer overflow issue in wasm-c-api reported by CodeQL
2023-02-13 13:55:50 +08:00
89c11c5361 Fix explicit casts and types in espidf_socket.c (#1945)
In the esp-idf platform, Xtensa GCC 8.4.0 reported incompatible pointer warnings when
building with the lwip component.

Berkeley (POSIX) sockets uses composition in combination with type punning to handle
many protocol families, including IPv4 & IPv6. The type punning just has to be made
explicit with pointer casts from `sockaddr_in` for IPv4 to the generic `sockaddr`.
2023-02-09 09:34:26 +08:00
3d44f11d19 Add gh api call for fetching llvm version (#1942)
Fixes #1877
2023-02-09 07:49:07 +08:00
3586911333 Fix minor typos in coding guidelines workflow (#1943) 2023-02-09 03:25:35 +08:00
251e63f3ec Enable specifying out-of-source platform configuration cmake file (#1941)
Resolve #1935, enable specifying out-of-source platform folder with
`cmake .. -DWAMR_BUILD_PLATFORM=new-os
               -DSHARED_PLATFORM_CONFIG=/path/to/new-os/shared_platform.cmake`
2023-02-07 22:10:59 +08:00
f3c1ad4864 Fix terminating stale threads on trap/proc_exit (#1929)
This is to terminate suspended threads in case an atomic wait occurs with
a huge or indefinite (-1) timeout, followed by a proc exit or trap.
2023-02-07 20:56:23 +08:00
ee1871d3f8 Enable iwasm --max-threads option for wasi-threads as well (#1939) 2023-02-07 13:46:41 +08:00