This phase I trial tests the safety, side effects, best dose and effectiveness of AB801 in combination with chemotherapy and immunotherapy in treating patients with cholangiocarcinoma or pancreatic adenocarcinoma that may be removed by surgery (borderline resectable), that has spread to nearby tissue or lymph nodes (locally advanced), or that has spread from where it first started (primary site) to other places in the body (metastatic). AB801 is a drug designed to block a protein called AXL. AXL is found on the surface of certain cancer cells and plays an important role in helping tumors grow, spread to other parts of the body, and avoid the immune system. It is thought to contribute to resistance against common cancer treatments such as chemotherapy, radiation and immunotherapy. In many cancers, including cholangiocarcinoma and pancreatic adenocarcinoma, AXL is overactive and associated with worse outcomes. AB801 inhibits AXL which may make cancer cells more sensitive to chemotherapy and allow immune cells to better recognize and attack the tumor. Chemotherapy drugs, such as gemcitabine, cisplatin, oxaliplatin, irinotecan, leucovrin and fluorouracil, work in different ways to stop the growth of cancer cells either by killing the cells, by stopping them from dividing or by stopping them from spreading. Immunotherapy with monoclonal antibodies, such as durvalumab and zimberelimab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Giving AB801 in combination with chemotherapy and immunotherapy may better treat patients with borderline resectable, locally advanced or metastatic cholangiocarcinoma or pancreatic adenocarcinoma.